devopsy.kinsta.cloud

Author: JP Morgenthal

DevOps and Organizational Debt

I recently published a blog on the Perficient Digital Transformation blog entitled “Is Cloud & DevOps Factored Into Your Digital Transformation?” In that blog I discuss the concept of organizational debt and its impact on your DevOps adoption. Here’s what I had to say in that blog:

If technical debt addresses too little investment in upgrading, poor vendor selection, bad platform selection, etc. then, many businesses have also built up organizational debt. Organizational debt could be over regulation, too many low-value policies, to much middle management attempting to hold onto their area of control, and too much governance. If these are not revisited often to ensure that they are continually adding value for the cost of reducing throughput, then that organizational debt will continue to increase backlog.

Interestingly, there’s a lot written about technical debt, but very little has been written about organizational debt, which can have a significantly deeper impact on your ability to continuously deliver than your technical debt.

In working with clients on their DevOps initiatives, we map the current value stream for a given solution area. What becomes apparent from these value stream maps is that controls and policies have been derived within organizational silos with no attention given to how that will impact IT’s overall ability to be agile and deliver more quickly for the business. Moreover, the owners of these controls and policies seem reticent to remove these bottlenecks even after being shown how it impacts business agility.

Here’s an anecdote from one client when asked why they aren’t using snapshots to speed preparation for QA testing. The suggestion was made in response to the client stating it can take upwards of a week to prepare the QA environment for a single round of testing. Their current approach serializes all testing and believes they need to perform a complete regression test due to the resource contention issues. Hence, they cannot test small changes easily and they cannot test often due to these limitations.

“The main issue is that we don’t control the application or database environment, so we need to prepare a QA release and then have those groups perform their duties in preparation of the QA environment for testing.”

Upon hearing this my mind summoned up past experiences working conferences at major venues, such as the Jacob Javitz Center in NY, where you can’t plug in an extension cord or run an Ethernet cable yourself, but you need to use the Javitz Center’s electrician because of union rules.

Is this what IT has become a set of unofficial unions that hold the business in a death grip? This is clearly a case of organizational debt that needs to be addressed.

Of note, I’m not admonishing the various groups for instituting their controls and policies. I’m sure they have emerged due out of need due to past grievances, failures or as part of IT management practices. Regardless, they are now acting as a bottleneck in implementing proven practices for delivering fewer features more often, which have been proven to increase trust and facilitate better prioritization of work.

The result of this organizational debt is lead times have grown to four months or longer to release functionality that is directly related to key business initiatives by the business for increasing revenue.

In his book, “Kanban,” author David Anderson discusses his recipe for addressing bottlenecks and constraints with existing teams. One of his steps is ‘attack sources of variability to improve predictability’. However, this is his last step because, as he notes, “some types of variability require behavioral changes in order to reduce them. Asking people to change behavior is difficult!” And, so it goes, many businesses never address this key constraint because the hurdles of organizational debt have become so burdensome to overcome. It requires a high degree of trust and political capital along with executive sponsorship. As shown by the anecdote earlier, even if the constraint is directly impeding other business priorities it is not easy to overcome.

I believe DevOps practices can help organizations significantly reduce their technical debt. I have seen and read about organizations that have been able to rapidly reduce their lead times while still refactoring and removing backlog. However, the goal of DevOps is to improve the entire flow, not just development, and a good majority of the release aspects of applications will require organizational debt to be addressed to enable improvement across the entire continuous delivery pipeline.

February 6, 2015
The Trouble With Enterprise DevOps

The literary world and beyond has been up in arms as of late regarding the use of the word ‘literally’. It seems that many are using the word literally when they mean figuratively. For example, if you say, “I literally split my pants laughing,” it would be plausible for one to look at your pants looking for a tear. Words are funny that way. When used to emphasize the action or outcome, some will take literary privilege. Enterprise is also one of those words that are used to convey emphasis often with disregard to the implied meaning.

When it comes to DevOps there has been a lot of debate about the modifier ‘enterprise’. Modifiers are supposed to add specificity to the noun it is modifying. Agreeably, enterprise is a horrible modifier for DevOps when used without additional modifiers representing scale. I have been, and continue to be, a big advocate of the concept of Enterprise DevOps. That said, I too have been remiss in my use of the word enterprise to further modify with the term ‘large-scale.’

I don’t remember exactly what triggered me to seek out if the word enterprise implied large, but when I did I was surprised to find that there is no implied meaning that enterprise means large and, in fact, can be modified with small-, medium- and large-scale to represent the appropriate size. As further evidence to the ambiguity of the use of the word enterprise, I worked for a company that divided their sales organization into enterprise and commercial accounts. Clearly, this organization believes that enterprise connotes large-scale since this group was responsible for selling to a specific class of customer that spent a significant amount of money with them. In this case, scale was identified by how much money the business spent with that vendor versus annual revenues versus number of employees. So, clearly, not only does the word enterprise not imply scale, but it also has different metrics that will determine the scale of an enterprise.

Merriam-Webster defines Enterprise as follows:

en·ter·prise

noun
\ˈen-tə(r)-ˌprīz\

: a project or activity that involves many people and that is often difficult

: a business organization

: the ability or desire to do dangerous or difficult things or to solve problems in new ways

Obviously, none of these meanings relate to size or scale, but two of the meanings do indicate difficulty. Let’s put that in the parking lot for later.

Why is this important? With regard to IT, I believe there is a tendency to throw around the word enterprise with no modifiers when we mean large-scale. For example, “Enterprise Software” is a term that is often used to describe very expensive and complex software applications that are designed to operate at the scale of the entire business versus the individual. Typically, the software described by enterprise software is found in mid- to large-scale enterprises where they can afford to acquire, configure and operate this type of software. Likewise, “Enterprise Storage” is a way to describe compute data storage environments for large organizations.

Likewise, “Enterprise DevOps” is a term that that is often used to represent adoption of DevOps practices in large, complex organizations with many different stakeholders and sub-organizations making it much more difficult to coordinate efforts around efforts like continuous delivery and continuous improvement.

Interestingly, and I believe accidentally, implementing enterprise software and adopting enterprise DevOps are difficult projects. So, based on our Webster’s definition, enterprise works as a modifier for these nouns. But there’s still the issue that for small-, mid- and large-scale enterprises the challenges and hurdles associated with DevOps adoption vary enough that the constraints and bottlenecks require different approaches to remove. This is especially true of large-scale global enterprises where development exists in four continents, operations data centers exist in multiple countries on four continents and the business operates in six continents with a comprised IT staff of 120,000 people.

I for one will be very careful in the future to define what I mean when I’m using the word enterprise in the future. If I’m talking about large-scale enterprises, which I usually am, I will make sure that early on in my writings and presentations I specify that when you see enterprise going forward I’m talking about large-scale enterprises.

January 23, 2015
Containers Are Designed For an Antiquated Application Architecture

In his most recent GigaOm article, “We’re finally headed towards autonomous, self-migrating containers for cloud-application automation“, David Linthicum discusses the value that containers, such as Docker and CoreOS Rocket, are bringing to DevOps and cloud systems management. For the record nothing Dave states in the article is incorrect with regard to the expressed value that containers provide for portability, however, there is an issue with the big picture view discussed for the role of containers in cloud application delivery.

Containers are predicated on the goal of deploying and managing n-tier application designs. By their nature, containers manage n-tier application components, e.g. database servers, application servers, web servers, etc., at the operating system level. Indeed, portability is inherent because all operating system and application configuration dependencies are packaged and delivered inside a container to any other operating system platform. Containers are…

–THIS STORY HAS BEEN MOVED TO CONTAINER JOURNAL–

January 13, 2015
DevOps Sometimes Means Kill It Before Fixing It
The public attention for DevOps is having a very positive effect on many areas of IT with regard to improving their ability to deliver for the business. Yet, I see little attention being paid to an integral part of this desire to change, which is to assess and baseline the current approach and then strategically select those areas where attention needs to be paid. Alternatively stated, many DevOps efforts are focused on improving current approaches without even determining if the current approach is appropriate for the business.

In their book, “The Lean Mindset: Ask the Right Questions,” the authors question the focus on productivity as an aspect of implementing lean or agile methods. This comes up quite often in discussion with customers regarding DevOps adoption. There’s a misconception that becoming more productive is the way to fulfill demand from the business. But productivity is not the right question or answer. Regardless how you approach the problem domain lack of infinite resources—money, time and people—the demand will typically exceed the ability to fulfill.

The way for IT to help the business achieve agility is to deliver the right set of features at the right time, which means the first step has to be looking at what you’re doing today and being willing to stop spending time and effort where there is minimal value or gain. The authors of “The Lean Mindset” have some interesting anecdotes around the realities surrounding this in enterprise settings.
- Politics – one company decided to financially incent development management to deliver agreed upon features on a behind schedule project even though it meant that testing would suffer and the final product would be unreliable. When later asked if they could have reduced the feature set and delivered fewer high-quality features, the team stated that it could have been done as less than 50% of the features were being used in the final deliverable anyway. The politics of delivering what was agreed upon was more important than delivering fewer high-quality features.
- Can’t Say No! – many IT executives find it difficult to push back on the business and, hence, end up attempting to deliver all that is already on their plate plus new requests that arise during the year. In many cases, much of what is already on their plate is no longer needed or provides minimal value to the business. Without a system of measurement around cost to deliver versus value delivered, IT cannot justify to the business the need to shift resources elsewhere to ensure optimal value.
For these reasons, among others, any attempt at DevOps needs to start with an understanding for where resources are being applied today and where they should be applied in the future. Doing useless stuff faster doesn’t help IT and it doesn’t help the business. It’s also why IT/Business alignment is a critical part of any DevOps adoption strategy. IT needs to work with the business in an open manner to select the most important features, capabilities, systems, etc. that will deliver value for the business. This also means IT is responsible for instituting the necessary systems to measure resource consumption and provide the business with the necessary data for them to make intelligent choices about where they want to spend their IT capital.

There’s a great Peter Drucker quote in the book, “There is nothing quite so useless as doing with great efficiency something that should not be done at all.” So, if your IT Operations team spends six weeks automating deployment for a system that really should have been sunset, that’s six weeks lost that could have been better spent on a more business critical function. Of note, this is not the IT Operations’ fault. If that system is not going to be sunset and has been the bane of their existence consuming time due to outages, then they have driven improvement. The failure is on IT management and the business having not taken the necessary steps to allow these resources to be focused on more important initiatives once again illustrating the importance of executive sponsorship for DevOps initiatives prior to tooling, automation, process improvement or organizational shifts.
December 4, 2014
The Conundrum of Adopting DevOps within Organizational Constraints

I wish I had a dollar for each time I’ve now heard, “we’re adopting DevOps, but we have to operate within the constraints of what the business is ready to accept.” Basically, this is becoming the battle cry of many enterprise IT professionals that are DevOps believers, but are stuck with the limitations of operating within a complex, political and immovable business structure.

One on hand, it is very pragmatic to approach adoption of new methods, techniques, technologies, etc. on a small scale and then expand based upon success. However, what is the result of a DevOps initiative in which existing organizational constraints and bottlenecks continue to remain in place?

If you’ve been in the business world for more than a few years you’ve probably already had the experience of having a really good idea only to come to the realization that implementation of that idea requires cooperation of another organization. For example, Marketing might have a great concept for how to acquire new customers, but would require sales management to provide incentives to the sales team for it to succeed. For purposes of this example, let’s assume that marketing has made its case to sales and sales rejects the proposal (many times there is a perception that the idea will be rejected limiting those from even trying; this is a worst-case scenario for any business). In this case, Marketing might attempt to operate around the need for the sales incentive and, ultimately, come to the realization to abort the plan or lower expectations for results.

Is anything more debilitating to morale then end up with these two outcomes as the only choices? What would you do if you were in the position of driving this for Marketing?

Often times in businesses there is no grievance committee. To steal a line from “A League Of Their Own”, “there’s no crying in business!” What DevOps is surfacing is that business is about relationships and managing cross-departmental and extra-departmental relationships is not something that many IT professionals have had to or want to foster. Ultimately, this will have a detrimental impact on DevOps adoption.

In most cases, DevOps requires that some leader within IT steps up with the idea to foster a frictionless ideation-to-operation lifecycle. By frictionless I mean the removal of as many bottlenecks and constraints for transitioning from the idea phase to a production-ready support application. Like Marketing in my earlier example, this may require this leader to “sell” their idea to the heads of IT Operations, Application Development, Quality Assurance, Information Security, and potentially C-level executives. A “No” response by any of these teams will result in the leader working around this limitation resulting in no change for the processes and work output of that particular group or attempting to do what they can within the organizational boundaries within their control.

Hence, business agility is tied to velocity of accepting change, which is further dependent upon individuals fostering a strong enough personal relationship–at a minimum one of mutual respect–to be willing to accept risk in support of a larger goal. I believe many would see this as a high-risk constraint on the business agility deliverable.

One way to increase likelihood of avoiding this trap? DevOps needs to be sponsored from C-level management from the start inclusive of the necessary incentives required to help drive the changes that will lead to more rapid adoption of DevOps practices and speed the way to business agility. There’s enough evidence now available to convince most C-level executives that a DevOps program is a worthy investment in moving their IT organization toward greater alignment with their business goals, producing faster, reducing operating overhead and improving quality.

As with most changes within IT, and DevOps adoption is no different, there is a requirement for bottom up support as those closest to the work need to believe, and there is a top-down requirement that provides the coverage, financing. motivation and accountability to the business to allow the change to occur.

November 20, 2014
Time-Sharing to Unicorns: The DevOps Problem Facing Enterprise IT

How did we in Enterprise IT get to this point? There’s a lot of talking and writing about the value of adopting DevOps but very little analyzing the problem domain we’re trying to correct. There are some schools that will say we can move forward without looking back, but I believe a key element of the dysfunction that faces many enterprise IT organizations can directly be attributed to their history and how they were formed.

One of the clear lessons we can learn from businesses that we call “unicorns”–termed this because of their mythical capabilities as an IT organization that can deliver high quality software very quickly–is that they have very little friction in the process from ideation to operation. Startups and greenfield initiatives are practically operating in a vacuum relative to the amount of friction they encounter compared to operations around entrenched decade-old legacy mission-critical applications. In retrospect, this should not be surprising given frictionless delivery is a mandate for many of these efforts and has been from the start.

You create what you focus on!

Some will say we don’t require the capability to deliver to production ten times a day (or even a year) for our SAP Enterprise Resource Planning (ERP) system. So they will conclude that perhaps DevOps isn’t right for our needs. This shows the skewed thinking that current literature on DevOps is fostering. It is not the need to deploy changes quickly into production that is at issue, but the ability to deliver business capabilities when they are required as quickly and with little impact as possible. Current processes around changes to the ERP are usually so fraught with friction that when the time comes to fulfill a need for the business so that the business can continue to compete effectively, it takes weeks or months instead of days or hours.

I researched how the division between development and operations was formed and interestingly, there’s very little history about this division of work. We can trace this division back to the early days of time-sharing systems in which the division of work actually represented the exact advice DevOps prescribes today. Deliver and operate a platform on which developers can build and deliver their applications. In the days of time-sharing systems, operations was not responsible for what developers put into the box, just ensuring that the box was available, secure, and backed up. If developers applications had bugs or crashed, it was the developers that were responsible for repair.

Then came the era of client/server. Application pieces ended up distributed across application infrastructure like a kid that dumped their Legos all over the floor. Managing these applications became as painful as stepping barefoot on a Lego as well. Developers didn’t work on the complete application, but instead worked on modules and these modules were split across various services each requiring their own levels of expertise to keep running in an optimal manner. The DBA was responsible for ensuring the database could meet transactional requirements and support data design and management, the network professionals dealt with protocols and traffic issues caused by these modules communicating with one another. Each layer of the application had it’s own capacity management and operational concerns. The only chance we had to keep the ship from sinking was to put it all in the hands of the one team that had access to all the components — IT Operations.

Things have not progressed much since this divergence away from time-sharing to client/server either. Today’s applications are more complex and more distributed than ever before. We have clusters of the same server working in unison to handle scalability exponentially increasing the complexity and difficulty for operations. A single application may now require anywhere from 5 to 25 servers to deliver the required capacity.

On top of all this we’ve never modified the division of work from the days of time-sharing and continue to push the burden for monitoring and uptime to the operations staff. We have turned IT operations into the Jigsawyer’s–someone who does jigsaw puzzles–and developers into the puzzle makers. Moreover, the puzzle is never the same; the puzzle makers change the picture all the time as they add new piece forcing the Jigsawyers to figure out how to retrofit the new piece into the whole.

While researching the history of IT operations and trying to figure out the path from computing’s earliest organization until now, I came across a great piece that really helps identify the problem we have created. So What Is A Deployment Really? by Robert van Loghem illustrates what it takes to move from ideation to operation with today’s application platforms. What I really like about this piece is the illustration that even if we want to go back to earliest division of work, it may not be possible. IT operations cannot simply provide a black box for developers to deliver their applications on and expect them to work as planned. Firewalls need updating, web servers need configuration changes to support the new domain requests and the security architecture must now allow these new requests to be routed appropriately based on rights and source.

Deployment is a team activity!

So, what have most of us in enterprise IT done? We broke down the tasks of deployment further into divisions of work so that we can apply the necessary level of specialization to ensure the application can be delivered to the end user at the cost of adding a ton of friction into the process slowing things down by a logarithmic factor when compared to our unicorn friends.

What is surprising is the number of pundits that put forth that the answer to this problem domain is empathy and culture change. Clearly, we have organizationally written ourselves into a corner because this is how large enterprises solve problems. Here’s a simplification of how this decision was most likely made:

Question: There’s a lot of issues attempting to monitor and operate these distributed applications and the developers don’t have the means or transparency into all the components.

Retort: Then who does have access and insight into all these components?

Answer: IT operations can see what’s running and has the tools to monitor the health, let’s give them the task.

At a time when IT was viewed as an expense versus a strategic advantage, there was no offsite planning to assess what the impact to this answer means to the business ten or twenty years hence. It’s solve the problem and keep moving because that’s what enterprises need to do in non-strategic areas. At the time this answer was formulated most businesses equated this decision with “which office supply vendor should we use?” Moreover, the leaders in client/server barely had any experience with IT operations, they were primarily comprised of application developers. Hence, there was no experience to guide businesses away from this decision. IBM, Unisys, Honeywell, etc. the vendors that were a prime source of guidance at the time were still leading with a time-sharing mentality.

This entire trip down memory lane was with one purpose in mind, figure out what we created and how it was created so we can consider the means to unwind it and provide a path for enterprise IT to move forward in an agile manner. The history provides the answer. We need to move away from the traditional time-sharing division of work that was formulated before the age of the Internet and realign along frictionless surfaces. What does that mean? Well, that, my friend, is an entire book that happens to be in the works! 😉

November 11, 2014
The Divisiveness Of The Term “DevOps Culture”

If the term DevOps has caught your ear of late and its caused you to start researching what it is and how it can benefit you, you’ve probably come across a number of dissenting pieces on the term “DevOps culture.” A recent tweet by Lori Macvittie of F5 (@lmacvittie) reiterated my thinking, “wow, this issue has some strong opinions surround it it!”

I’d be remiss if I didn’t mention that I have on numerous occasions stated my own issues with the notion of culture being an instrumental component of DevOps adoption. Having been an executive in a few startup tech companies in my past, I remember, and still smart from, the various executive team discussions about company culture. We spent hours–wasted hours???–discussing what kind of company we will be and how to build a culture that would turn us into a market leader only to have all that work undone by underhanded backstabbing and small company politics within the executive team itself.

Hence, I’m skeptical about the ability to truly foster culture anywhere. Culture is something that takes holds as a factor of the individuals involved and how they treat each other, which is a direct result of many other corporate led initiatives, such as opportunity to be recognized for their work, stability, compensation, etc. So, if it’s that difficult to foster a culture in a small startup, what is likelihood of fostering culture in a large enterprise, where so many in IT struggle everyday with great frustrations and an inability to progress through backlogs and fragility?

There’s a great Twitter discussion that includes Damon Edwards, CEO of DTO Solutions (@damonedwards), John Willis (@botchagalupe), Matt Stratton (@mattstratton) and myself on the topic in which Damon discusses how he and John Willis came up with the CAMS acronym–Culture, Automation, Measurement & Sharing–as a means of avoiding partial solutions.

In this thread, I specify that enterprises thrive on tangible, repeatable efforts. For me, attempting to explain the benefits of DevOps to the business to an IT executive needs to be rife with tangible and measurable efforts so that the IT executive can have a high threshold of confidence that if followed will lead to the expected results. In my opinion, culture does not belong in this conversation for the sole purpose that if the culture cannot be changed, the entire effort will be suspect and, thus, most likely not attempted.

In response to one of the tweets in this thread, Damon states:

Damon has been working with many large enterprises now on delivering benefits of DevOps, so I believe this statement is reinforcement of my earlier theory that culture is a big turn off for getting IT executives to invest in adopting DevOps approaches. Organizational dynamics is something that is a bullet point within the roadmap that must be dealt with and the IT executive will recognize that no transformational efforts can ever be successful without possible need to shift people and roles.

In speaking with Edwin Miller (@edwinmiller), CEO of 9Lenses, a company that provides a platform for analyzing organizational dynamics, there was agreement that culture is important, but it is very much a leadership issue. Without strong leaders that can admit fault, don’t fear not having all the answers and can incite the passion of workers any attempt at culture is a folly. The opportunity to have real transformation in IT will start with hiring IT executives that understand the changing landscape of the digital economy and realize, just like the automotive companies did in the 80’s, that middle-tier executives may have no perspective on issues where line workers have great insight into how to change things. That is, in order for things to really change in enterprise IT, it will require leaders that recognize that the aggregate knowledge of all in IT will be required to identify problems and eliminate them.

With regard to the perspective of those in favor of DevOps culture, it seems there are various perspectives on what is meant by the word culture. This blog entry by Jeff Sussna, “Empathy: The Essence of DevOps” has become a beacon for many on how to proceed by focusing on fostering change through shared experiences. This article, “DevOps Culture,” by John Willis discusses culture from the perspective of behavior modification, where behaviors are essentially how the organization responds to a given stimulus. This latter piece really is about problems caused by the overlap of people and process, which is an interesting point given that we talk about IT being about people, process and technology as if they were independent silos when what we really are talking about is the need to fix the points where these three elements overlap.

Frankly, stepping back from the fray for a moment, I find it odd that opposition to the term DevOps culture causes such fierce debate and in some cases mob-like responses in various forums. For me, I’ve presented a logical reason why the term can be seen as negative or at a minimum offer no value to entice investment or gain traction for taking action to work on the changes that are often discussed around companies practicing DevOps. Other than “it’s critical to DevOps success” messages, I’ve seen little from those pro-culture advocates as to why we should expect businesses will change something as esoteric and intangible as culture in order to succeed in fixing what we should all view as ineffective practices. Without meaningful changes in IT management to hire leaders with the aforementioned attributes, talking about culture change is akin to changing the oil filter in your car when the spark plugs are shot.

October 2, 2014
Being Agile is all about controlling the flow

For businesses, Agile is the new fashionable initiative as was E-business was in the 2000’s, Six Sigma was in the 90’s and Total Quality Management (TQM) was in the 80’s. If we look back across the spectrum of these initiatives, we need to examine how many of these initiatives met the goals the business was looking for and, more importantly, if they were fully adopted, remembering that many were derailed, did they deliver the value the business was seeking?

Of course we cannot answer this question with generalities—it’s a spectrum. For some businesses, these initiatives were red herrings, some did not follow through to completion, but certain changes adopted as part of these initiatives had a very positive impact, while others succeeded wildly. It’s a Harvard Business Review case study as to why a business ended up at the point on the spectrum that they did.

Clearly, one fundamental aspect that is consistent across all these initiatives is the ability for a business to embrace change. In a previous blog, “Cloud Computing, DevOps and Business Velocity,” I addressed the issue of business velocity—the rate at which an organization can embrace a particular disruptive technology change. Underestimating your business’ ability to adopt a new technology or process will most likely have a strong negative impact on the outcome. In a recent conversation with a client, the non-IT business areas has been very successful at adopting lean practices, but IT short-circuited shortly after starting. It didn’t outright fail, but after a strong start, they took a break from which they never started back up. Without a post-mortem, we don’t know why this specific initiative fell flat, but we can intimate that if something is difficult and feels unnatural, we’re more easily able to focus our attention elsewhere—think dieting and exercise programs in your own life.

This seems to be the decade of Agility. Businesses want to be more dynamic and fluid. They want the organization to be prepared to respond to market changes more quickly, introduce new products faster, and respond to problems that occur more rapidly. To accomplish this goal, the business will have to let go of some of the ways that they do things today. They will have to re-engineer processes to be more compose-able and componentized so that they are less static. For example, a manufacturer goes through a lot of planning to deliver a new product to market. They go through steps for design, prototype, test, source, manufacture and distribute. The entire cycle may take upwards of a year and is highly automated once it reaches the manufacture stage. Tweaking the product between batches would cause significant delays and pain because the process is static. It effectively amounts to restarting the entire cycle. So, what if they discovered that by simply elongating the body by 2 cm the product was far more useful their intended audience? In this case, agility might equate to being able to complete the design-to-manufacture process faster.

In the previous example, this latter step of being able to tweak the product is called continuous improvement and is a key element of fostering agility. If a product isn’t selling well in the market, many companies will just abandon the product completely due to the costs associated with making the product viable. The reason being that most manufacturers consider that each product is a project and when it reaches the manufacturing stage it moves under the control of operations. That is, the creative team that includes marketing, finance, and other stakeholders view their position as “completed” with regard to getting that product to market and they then move onto other projects. With this mindset, there’s no room for the group to review the process, think about how it could have been done better and then iterate with the improvements incorporated. Hence, they just keep taking their “this is the way we do it” to the next project.

Being agile means being able to control the flow. There’s no way to change how products come to market if there’s a culture of accepting finality once a product has reached a certain point in the cycle.

The other aspect of controlling the flow is managing how products and services enter and exit the development pipeline. I spend a lot of my time helping IT organizations unlock their potential to deliver by helping them to understand that there’s only so much that can be done to improve managing the work-in-progress (WIP). The only way to make gains beyond a certain point is to enable transparency with those that are responsible for inserting work into the pipeline or who are demanding that work completes and exits the pipeline at a certain time.

The following two figures express well the reason transparency is a requirement. I believe most people would agree that managing production is a far easier task in Figure 1 than Figure 2. In Figure 1, there is one input and one expected output. Managing WIP for this pipeline is a matter of optimizing just the process for completing that one deliverable. In Figure 2, there are many more variables at work. There could be contention for resources within the pipeline to support the multiple inputs.

Likewise, there may be a request to reprioritize what gets completed when to meet certain business goals. Perhaps a particular report is required urgently to support a key business decision. I personally dealt with this very scenario when the business I was working for years ago was suddenly engaging in acquiring another company. All focus was transferred to producing the needed financial statements and everything else was put on the backlog. In that particular case, the CEO and CFO were the customer, so no one questioned the delays to their projects, but for many IT organizations reprioritization requests are never that clear making it very difficult to manage the control of the flow. Hence, transparency and communication between those consuming the services of the pipeline and those who own the control of the flow of the pipeline is an imperative for agility.

Like TQM, Six Sigma, and E-Business, Agility will require changes to the fundamental way that a business operates. An analogy of this process can be as benign as a person recovering from knee surgery or as complex as recovering from a stroke depending upon what the business wants to accomplish and how they are organized currently. Along the way, people will most likely be the biggest hurdles to success.

When change of this magnitude occurs, employees tend to become very myopic of their own situation. To mitigate this risk, it is critical that changes are reviewed with employees prior to change, during implementation and after the change is enacted. Management must gauge if employees are struggling because the complexity of the change or because they personally don’t agree with the change. The former can be overcome with education and support while the latter should be dealt with quickly through removal.

Businesses can succeed with their Agility initiatives given they identify and mitigate key risk factors prior to starting and are willing to commit to looking critically at the way they have been operating the business. In agile software development, developers are trained to think about ways their code can break and then write unit tests to ensure that the software is sustainable in light of these possibilities. Creating an agile business means looking at ways to ensure sustainability in light of market changes, new competitors, supplier changes, and changing customer demands.

August 28, 2014
Reconciling DevOps Messages: Notes From DevOps Days Minneapolis

Last week Minneapolis was host to Major League Baseball’s All-Star game, but when they were done cleaning up after the game a group of IT professionals got together to exchange ideas and share experiences about DevOps. It was DevOps Days Minneapolis. The crowd was a highly diverse group of individuals including management and engineers, uncharacteristically roughly equal numbers of men and women, and theorists and practitioners. The mix led to some really great exchanges and discussions with a repeating pattern of emphasizing the cultural significance of DevOps.

This is a passionate community. People that are participating really want to improve delivery and operations of solutions for their companies. However, the lack of the definitive in DevOps means there’s a fair amount of “you’re doing it wrong!” being leveled at others in the community. I myself may be a guilty of doing this a time or two. Here’s just some of the rising debates that emerged across the presentations and unconference meetings.

Philosophy vs. Structure

There was a lot of philosophical discussion. For example, Dan Slimmon talked Conway’s Law—described by Slimmon as “Organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations”—as a force that influences the need for DevOps. Meanwhile, Jeff Sussna presented on how promise theory and its role in delivering continuous service quality.

Likewise, we heard from the team at Target. Heather O’Sullivan & Ross Clanton told us about the pragmatic way they are introducing DevOps across both Dev and Ops demonstrating that it’s possible for large enterprises to adopt DevOps strategies and apply them to deliver faster with higher quality. And Ben Hughes of Etsy described how security fits into the DevOps landscape.

Hence, a good balance of what some people are doing while also a fair amount of looking outside the IT universe for examples of ways to improve.

Culture vs. Process

Incorporated into the ignite talks and some of the Open Spaces there was a heavy dose of this thing called a DevOps culture. Katherine Daniels’ presentation discussed the transformation from system administration to “sparkly DevOps Princess”. Of course, over time we have also heard about Unicorns and horses as a way to describe what looks like a DevOps culture.

But culture is such an overused word. Google reports one definition as, “the attitudes and behavior characteristic of a particular social group.” Are we to believe that attitude and behavior are the most critical element to correct the long standing issues of developing complex systems in a fast moving business universe? Certainly it helps, but there’s a lot to be said about a group of people coming together to accomplish a mission even if they don’t share common attitudes and behaviors. So, perhaps, the ways that we leverage tools and process, such as how you can you leverage automation technologies to remove redundancies and control configuration drift is itself a culture?

Lean IT vs. Dev+Ops

Which leads to the most interesting aspect for me. I had the opportunity to actually discuss some of this with Patrick Debois and asking how he reconciles the Lean IT/Agile approaches put forth by Gene Kim et. al. in the Phoenix Project and the foundational aspects of DevOps, which is agile for system administration.

The Phoenix Project is often considered the seminal book on DevOps, but it presents a much bigger view than the concept for how agile development processes can be applied in operations. Automation, Continuous Integration, Continuous Deployment, etc., these terms are synonymous to saying improvements on the shop floor in a manufacturer is the same as Lean Manufacturing. Improving Dev+Ops is important, just as improving shop floor processes is, but if you don’t change the way orders are introduced into production, how they are prioritized to meet customer needs, and limit bottlenecks before it reaches the shop floor in addition to controlling them in production, all the automation in the world still won’t guarantee higher quality or more throughput.

All-in-all, the Minneapolis team put on a phenomenal event. And, as an organizer of the forthcoming Chicago DevOpsDays, I am looking forward to being able to deliver as valuable an experience to our attendees.

July 23, 2014
Infrastructure-as-code, new rules for the old game

There’s been a lot of discussion regarding DevOps and its role in fostering quality and continuous improvement through increased levels of automation. In turn, this automation is fostered through emerging infrastructure and software architectures that are exposing control plane and management programming interfaces, which has been dubbed in many circles as Infrastructure-as-Code.

This movement allows development and operations personnel to have real-time interactions with the underlying environment—the network, servers, operating systems, and storage—that are used to deliver applications and data. It also means that as the environment becomes more programmatic the business must treat this this effort as a first-class application development process and ensure that the corresponding assets follow common coding practices and meet with a consistent and sustainable architecture.

Those that are unfamiliar with the activities of system administrators and data center operations may have an ill-conceived notion after reading emerging literature on DevOps that operators have been running the business on the backbone of purely manual processes. Even as a life-long IT person, until I started working with infrastructure and operations as part of my role, I was unaware of what happens behind the great curtain that keeps the digital bits flowing through the organization.

The truth is that operations and administrators have been developing scripts for years to automate routine tasks. These scripts have typically been used to dynamically generate or update configuration files that were then applied after a restart. This change to Infrastructure-as-Code is significant because now these configuration changes can be made on the fly and are often immediately acted upon without having to take the system offline. If you’re now putting your hand over your mouth upon the realization of the potential disastrous effects this can have, you understand the impact of this power.

Thanks to Software Defined Networking, hypervisors, and even Software Defined Storage, an operator can programmatically change the entire environment underlying the application without having to notify the application of this change. Architecturally, this is a wonderful thing. Pragmatically? Well, there’s a lot of bad code out there that makes assumptions about the underlying infrastructure and is about to be exposed when migrated to these modern infrastructures.

Hence, building, testing and deploying infrastructure control applications must become part of the overall Software Development Life Cycle (SDLC) and begin to be built, tested, deployed and managed using the same controlled techniques that are hopefully being used for any applications that are being developed by application engineering.

You may be asking, “Wait, doesn’t that mean my QA environment must have the same infrastructure components as my production environment?” To wit I respond, “Maybe!” In some cases this may be advantageous for ensuring that production deployment can support continuous deployment processes, which will ensure limited human intervention from the point of user acceptance through deployment, which will, hopefully, lead to limited opportunity for production failures. However, because we’re moving to a more programmatic infrastructure many vendors can also supply virtual instances of their infrastructure code for purposes of testing and this may be satisfactory for your needs.

Of note, there are many businesses out there that do not develop their own software. They primarily use Commercial-Off-The-Shelf (COTS) applications. Thus, these businesses may not have an established SDLC practice. Likewise, these businesses may not believe DevOps is relative for them since they do not do “development”. If your business falls into this category, the move to Infrastructure-as-Code and Software Define Data Centers should be perceived as a driver for DevOps adoption as well as formally adopting one of the well-defined SDLC approaches. Even if these components won’t impact you for a few years until your next refresh cycle, the advantages DevOps brings around improving quality and consistency in delivery is of definite value to your business now.

Infrastructure-as-Code is a significant change in capability that operations has for delivering compute and storage resources to the business. It will allow them to have greater insight into the hardware allowing them to do predictive maintenance, migrate applications to various resources based on demand, and tune the overall environment to meet the business’ needs at a given moment. In short, it will allow IT to deliver compute services with a high degree of availability and security. It will also expose shortcomings in existing applications that have made assumptions about the underlying environment on which it is operating. Thus, if operations and application engineering are not working together and the entire application environment inclusive of the infrastructure control apps is not tested and deployed as a unit, the likelihood for failure will be high.

June 30, 2014