devopsy.kinsta.cloud

Tag: enterprise devops

Dying for Intelligent Calendar Awareness

What would we die for in an integrated DevOps tooling solution—something that would make our operations partners stand up and cheer? Intelligent calendar awareness.

The most fundamental construct of any release orchestrations (RO) tooling software is the calendar. RO software, no matter who the vendor is, uses a form of the calendar, to outline what tasks will get performed in what order from beginning to end. Better RO software allows for parallel tasking, but all RO software uses a calendar. So the most fundamental use of a calendar construct is focused on “when” tasks are performed and completed. Adding intelligence to a calendar construct adds three more dimensions or characteristics to this baseline feature.

The Intelligence of Ownership

First is the concept of ownership. Instead of just marking a calendar with the dates of specific tasks, or clusters of tasks, the addition of ownership marks the calendar with “events” that are “owned” by specific teams of people intended to perform groups of actions or tasks. As an example, instead of just marking the calendar that on April 27 this year, with the specific tasks we need to run a mini-release to upgrade our Oracle database from version “x” to version “y.” With the addition of ownership, we could mark this date as running an “event” of an Oracle database platform upgrade. This new event would be “owned” by the database update team.

The block of time the team outlines for the Oracle upgrade “event” would then take on two additional characteristics for teams in general: inclusionary and exclusionary implications. As an example, the time set aside for the tasks to upgrade the Oracle platform becomes inclusionary for the owning team. Members of this team would be allowed to schedule tasks during this block of time, ideally intended to facilitate this goal of upgrading the platform.

Other teams are (by default) excluded from performing actions during this block of time so they don’t impact what the database team is attempting to accomplish. When other teams pull up their respective calendars to schedule some new task or event, the April 27 block of time is greyed out or presented as unavailable because the database team owns an event listed in that block of time. For the database team the time is inclusionary, for all the other teams the block of time is considered exclusionary.

The benefits to you the customer are immediately visible. By adding ownership to the calendar construct, events could be scheduled in such a way that teams do not inadvertently “step on” each other (i.e., attempt to schedule tasks during the same window of time that would adversely affect the other). It also allows you the customer, to view a dashboard that shows the “kinds” of events that are scheduled throughout the week, month or quarter, and who is doing what.

To avoid being too strict with this idea, the RO tooling vendor need only add concept of granting exceptions to requesting teams by the owning team for events that would not impact their efforts. So as an example, if a different team wanted to work on tooling the SQL Server platform during the same block of time that Oracle platform was being updated, and the servers were different anyway, there is no reason why that team could not schedule a similar event of its own at the same time. If ownership is the only concept added in the RO tool, an exception would have to be granted by the Oracle team, but this would be costly in terms of administration. It is for this reason we introduce the second main construct in the intelligence of calendar management.

The Intelligence of Component Awareness

The second main concept of intelligent calendar management is component level awareness. This dimension of awareness is based on identifying specific “technologies” as resources or components that are part of the environment you operate at your company. It doesn’t have to be a ground-up exercise, and if you prefer one of those, you could use a standard discovery tool to keep it up to date. But the more basic idea is to simply identify the fundamental technology components you use in your world.

So as an example, I could identify that this company uses Oracle databases (perhaps also tracking the version if there are multiples of it), SQL Server databases, the type of servers it uses, the operating systems, the middleware, perhaps a mainframe platform and the network fundamental technologies, etc. It is highly likely that there are groups of people who are uniquely identified in this company that manage these specific types of technologies. This makes the associations easy for event ownership. The Oracle guys become associated with the Oracle technology components but not with the SQL databases, etc.

Having teams associated with the components they manage allows for better parallel event scheduling. But this is only the first level of intelligence. The next level is to track, by application, the underlying technology components they utilize. This process is done during the build and deploy automation anyway, so the effort is nothing more than what is done today. The beauty of having this spider web map, or this map of technology component dependencies, is that it adds a new level of capability to scheduling in our calendar.

At this point, scheduling an Oracle event does not preclude the SQL team from scheduling another one during the same time window, since those two technology types don’t conflict with each other. Both teams would see open calendars on the same day due to this intelligent calendar awareness of component interdependency, or lack thereof. This dimension of intelligence precludes having to grant needless exceptions to owned events. The only requests for exceptions would come from users or teams that actually would impact the upgrade event, and they could be evaluated for legitimate need and potential real impact.

Let’s return to our Oracle platform upgrade event our mythical company scheduled for April 27. With the addition of component-level awareness, when this event is scheduled by the Oracle database team, every application that uses Oracle is “automatically” notified that this time window is now off-limits for its development or usage activities. This limitation could include automatic notification to application development teams that they could not deploy new code during this window. Or the notification could be expanded to notify users of the impacted application(s) that this list would be down during this window of time. Applications that do not use Oracle would remain unaffected and not notified.

Imagine what this kind of automatic notification could do for both development and the user base of the technologies you operate, all coordinated automatically through the RO tooling you use today in DevOps. Then think about the possibilities from here. The Oracle team could look at a block of time again and examine whether scheduling on April 27 would have a high impact on other teams, or not, before it finalizes it. If the impact to others is high, perhaps the team picks a different date. The possibilities become endless.

The Intelligence of Scope

The third and final architectural dimension to add to calendar intelligence is the concept of scope. Scope is a simple hierarchy that you could define about your company by logical business subdefinition. As an example, you could outline the scope you define as: global (entire company), business unit (listed by name), product unit (listed by name) or departments (listed by name). Just using these kinds of simple scope definitions begins to insert an impact assessment into the event scheduling that takes place.

As an example, events could be scheduled with a global scope, owned by the change management governance body, that excludes a 10-day window because of a corporate audit by outside regulators. Now, when any development team attempts to schedule deployments or releases, for example, their calendars are automatically greyed out during this 10-day window because of the corporate governance event. Exceptions would only be granted by that governance body. In this case, this prevents accidental changes occurring during the middle of an audit.

If the example cited above is too broad, and the mythical company only needs to limit an event to a particular product line or further down to a particular department, the “scope” characteristic allows them to do so. This final dimension to intelligent calendar management does require a higher level of sophistication from the RO tooling vendor and a higher upfront investment in the administrative costs of providing DevOps services. But for an enterprise-class installation, this is an investment that can take the chaos of corporate communication where it comes to the impact of change to a systemic nirvana of communication on demand.

The Impact to Your Bottom Line

So why outline an architecture that most RO vendors have yet to implement? If it is candy I created that you can’t have yet, why discuss it? Two reasons: First, when you begin to examine the cost of running your DevOps set of services, you will find that, like many other enterprise-class efforts, it is the administration costs that are heavy at the outset and increase over time. The effort it takes to manage “communication” and avoid having interdependent technologies “step on” each other is no easy task, particularly when we have moved from weekly deployment windows to potentially daily deploys to production across large app portfolios or complex legacy hardware architectures.

Second, for those of you who still maintain data centers, consider how much time you “waste” if your Ops teams continue to require a 12-hour to 48-hour window over the weekends to make updates to their own platforms. Going to an external cloud provider is a way out of this, but for organizations that are unable to fully do so, the weekend downtime remains a significant issue. DevOps will create desire in the Dev teams to want to use this traditionally Ops window for their own purposes as well. Once DevOps has been fully embraced, there is as much a need to automate communications and intelligently ensure we do not step on each other as there ever was. DevOps accelerates this need, as it accelerates the speed of innovation itself.

The proper tooling, or rather the tooling that is thinking about intelligent architectures that will greatly reduce your costs and your overhead and ensure consistency in your world—these are the features (and the thinking) you need your tooling vendor to embrace. The alternative is having to pay your labor base to manage it another way. If your leadership and your vendor partners are not considering these issues, they should be. If you need help, feel free to contact me (I am still looking at the moment). The goal of DevOps to automate process should not just be about moving code; it should extend to how we administer, communicate and inject governance as well. Intelligent architectures accomplish these goals, and any vendor could be there first, if they wanted to.

To continue the conversation, feel free to contact me.

May 16, 2016
The Beauty of Inheritance Architectures

Spoiler alert: This article is not about inheriting your great-grandparent’s estate. It is rather about how embedding the principles of inheritance into the architecture of your chosen DevOps toolset can yield exponential results. And yes, I realize that you, the customer, have little to do with what a vendor builds its tooling architecture upon. However, you, the customer, can vote with your feet, to search out products that offer features such as inheritance, rewarding vendors that have it and punishing those that do not. The simple question to address here is, Why should the customer care?

Like anything that can grow, sustainability on scale is a key concern. How much will it cost me when I am maintaining 6,000 applications on this particular DevOps toolset, as opposed to the six applications I started with? You can also infer maintaining a higher number of technologies as having the same need. For example, if I need to maintain three different types of databases, two different middleware, four different hardware platforms on “x” different versions of operating systems … you get the idea. The larger the application portfolio (in sheer numbers of apps), or the more technically complex the portfolio (in sheer numbers of technology components in an app), eventually will drive you to worry about how easy it is to maintain these applications within your chosen DevOps toolset.

The capabilities of your chosen set of DevOps tooling will govern the costs of both the initial adoption of your services as well as the ongoing maintenance of the application portfolios you serve. It is hard to sell the value of DevOps to the business when it begins to see a one-for-one increase in DevOps engineering staff for every new application or technology venture under consideration. However, one of the chief culprits in driving large DevOps support staff organizations boils down to the lack of an inheritance architecture in the tooling that was originally selected.

Setting a Common Language

So before we discuss how inheritance will benefit us, first let’s agree on a common vernacular of what an “application” is. When you say the word “app” to someone, you get a wide variety of responses on what that means—from something that sits on your phone to a cluster of things that sit on the mainframe. Let’s begin by examining what an “app” is by the effort it takes to onboard and then maintain it on our DevOps sets of services (regardless of platform).

Consider for a moment the hierarchies by which we could describe software constructs. It is relatively easy to discuss how software works in terms of hierarchy. First, at the lowest layer, we could describe having multiple technology components (or the lowest level of a particular piece of technology that must be addressed in a build). Next, assembling these components together forms the picture of what a single application looks like. Finally, assembling multiple applications that have a common symmetry or common purpose together form a system or series of apps.

An imperfect but quick example could be the Microsoft Org Chart component of its Office application(s). For purposes of example we will argue that Org Chart, Clip Art Manager and Equation Editor are all components of the application Microsoft Excel. To build a new version of the application Excel, we would have to deal with each of these individual components that, once assembled in our build, would create a new version of Excel. When you combine more of the apps—Excel, Word, Outlook and PowerPoint—you get the system, or series, called MS Office. (And, yes, I know this is a less than ideal example, but it is meant only to illustrate a point.)

Assessing the Work Effort

The reason for discussing software in this way allows us to begin to discuss the level of effort to automate it. By addressing the work effort to automate each component (by type), we can begin to estimate how long it will take to complete “the automation of software.” For example, if my homegrown application has a source code component, a configuration file component, a database component and a middleware component, then I have four components I must automate (or include in the build and deploy process) to assemble a new version of my application.

This gives us a “least common denominator” language for assessing how much effort it takes to automate a given app. I have seen most common applications have only two to three components. But I have also seen several apps that, due to their unique architecture and construction, have 30 to 50 components a piece. Obviously, it is easier to automate an application with only three components than one with 50 components. The other factor we need to apply on the ease of automation is whether I have encountered this particular component technology before in my prior automation efforts.

Let’s say, for example, that the application I am estimating contains Java component(s). If I have encountered Java automation prior to this, I should have constructed a template for Java type component automation that I can use for a situation just like this one. In this situation a familiar technology component is far easier to automate than a new one (or, rather, new to me). However, if this is the first time I have ever seen Java, then it may take me a bit longer to construct the template I want for this app, and for future apps I encounter, that also use this technology type.

The Beauty of Inheritance

Here is where inheritance begins to play its critical role. If my DevOps-build construction tooling includes inheritance, then I could construct a “master” template or framework for each type of technology as I encounter it. A master template for SQL databases or Oracle or DB2, as an example. Or, I could make a master template for WebSphere or SharePoint types of technologies—one master for each kind of programming language we have in my shop. You get the idea.

Having done this allows me to have a limited number of master templates, in which I can maintain all the critical steps and processes my company wants me to do for a given technology type. If different business organizations or development organizations need a variation, they can create a “child” of my master and vary it according to their own needs. As long as the linkage is maintained to my parent/master template, I can make the big, overarching changes driven by corporate mandate, and the business unit can continue to vary items to their unique needs.

Having inheritance in this instance allows me to roll out changes to an entire organization, such as, for example, a security patch or process, for a given technology by simply adjusting the master template of that component. Inheritance automatically flows my change down to all the “children” of my master and possibly to all their “children,” should the need arise. The alternative is having to go through the specific automation of each application that has the targeted technology component and updating it manually, hoping I don’t miss one, and hoping the steps I may need to adjust do not break the old automation instead of alter it the way I needed.

Imagine having a WebSphere subject matter expert able to adjust every client application that uses your corporate installation of WebSphere by simply adjusting one master template. Or having a network guru, or database expert, be able to do the same for all applications that use their particular technology component all at once. Inheritance preserves the real-world need of having limited alterations of the same technology type within the same company, flowing down changes from parent to child.

Having a one-size-fits-all is simply not practical. Using copy/paste requires repeating updates to each impacted template or framework manually. But having the ability to create offshoots from a main, and not lose the connection to the main, allows corporatewide alteration from a central point of control. The costs of maintenance or administration are formidable; inheritance can reduce them. This allows a much higher ratio of applications or technology components supported by a much lower number of engineering staff.

A Wider Impact for Inheritance

Inheritance is keenly useful in release orchestration software as well. Most vendors offer the cut-and-paste approach: you copy the last the release you did, alter it slightly and use it again. That might work for some organizations. But having an inheritance feature creates entirely new dynamics on how a release is constructed for a given line of business and how corporate governance can be implemented with minimal effort. In this case, a corporate master release template can be constructed, and child releases can inherit from the master all the main governance required without risk of omission.

I realize inheritance architectures might not be at the top of your list when evaluating DevOps tooling solutions. But I would submit it is a strategist’s job (if you need one, let me know, I am looking 🙂 ) to point out how long-term sustainability can be greatly benefited by this architectural feature—even if the vendor would love nothing more than to steer you away from it. If you already have committed to a tooling solution, you may want to bring this up at your next vendor features performance meeting. If not, you should at least consider what having it means to your internal cost structure, and what not having it will add to the true cost of ownership of any proposed solution.

To continue the conversation, feel free to contact me.

May 11, 2016
Humans in the Machine of Release Orchestration

Let’s face it: Most organizations are still struggling to achieve the Nirvana of full gated automation. At first, we automate the core of our DevOps services, the build and the deployment disciplines. Over time, we realize that a templated or framework (or pick your similar terminology) approach toward automation allows for scale as well as predictability, lowering costs. It is here where the lion’s share of variability can be reduced and scalability achieved.

The next culprit to hit the radar is usually test automation, but this tends to be where the rubber meets the road for the level of work it takes to automate process. So generally minimal work gets done, ascribing to the 80:20 rule that 80 percent is quickly attainable and should be sufficient, the remaining 20 percent is just too hard. But having gotten this deep in the DevOps maturity scale, release orchestration (RO) is usually next up in our continuum of services.

The automation of release orchestration is a different animal than the core of build, deployment or even test automation. Release orchestration is more about automating our process of change management. Invariably the spreadsheets or SharePoints of yesteryear are intended to give way to our preferred vendor’s methodology of reinventing the release wheel in their proprietary toolset (albeit with better results expected). But usually this is the wrong approach as well.

Changing Human Thinking About RO Construction

As I continually repeat, DevOps is about thinking differently about “how we innovate.” (If you need more strategic leadership help in this regard let me know, I am looking.) Consider that our spreadsheets and SharePoints, while impressive, constricted us to thinking about release and change management within the architectural constructs or limitations of the spreadsheet and SharePoint tools. They were task-oriented, less interactive, top-to-bottom lists of what we thought we needed to do in any given release. If we were to truly go back to the drawing board and thought about creating a release without those constraints, or just with the idea of parallel tracking, what might we accomplish?

Digesting the release process with a lean mentality breaks down large sections into more manageable subsections. This reduces risk, and increases speed, particularly if it is coupled with parallel execution. If there truly is a sequential dependency in our tasks, let’s call it out, but when we do, we must ask ourselves whether that dependency is a go/no-go for all activities, or for only some of them. If it is the former, so be it; if it is the latter, then perhaps we can adjust our sections to account for this and run more of them in parallel. Just implementing a parallel mindset can radically reduce the time it takes to run a release from end to end. But we must also ask ourselves, What other key features of any preferred release orchestration toolset can alter how we think?

Changing Human Thinking About RO Execution

The other axis upon which to digest release orchestration is in its real-time interactivity. Consider for a moment that some tasks either cannot or will not ever be fully automated. Some tasks will remain in the province of a human to fulfill. For example, (and I cannot imagine why, but humor me), if we needed to install additional hard drives in bare metal servers during a release, the physical installation would always require a human to do so. With old release orchestration technology we must wait for shared spreadsheet files to be updated by the engineer when he/she completes this task (or any other manual task example you can think of).

With new mobile release orchestration technologies, the engineer could mark them complete (or percentage complete) in real time on his/her mobile device while still in the data center, standing in front of the hard drives/server. How much time does this save, in pure walking to his terminal, logging in and updating a file, emailing it to the change manager, merging it with the current one, etc.. A simple swipe and done. The engineer is now able to provide a great deal more information to the team in real time, and with minimal effort. This gives real-time visibility to the release manager such as has never existed before.

It can also grant them a level of direct interactivity. Imagine the hard drives’ physical installation has been completed, but manual testing is failing. If the engineer has no good way to share the information with the team, risk for overall on-time completion cannot be properly assessed. Proper tooling avoids the scenario where change and release managers are waiting on a task, and drill-down activity is unavailable (picture a spinning icon, rather than a percentage complete on specific task item). Add the dimension of red-yellow-green to track the completion of tasks, whether parallel or not, and we have a dashboard that calls our attention to the problems impacting a release in real time. This is far less possible with older RO technologies until somebody checked their watch and started making phone calls.

Changing Human Thinking About RO Valuation

Now every organization dreams of a release containing nothing but virtualized full-stack deployment for every app (even the legacy ones), where everything from bottom to top is automated during the deployment process. However, some of us living in the real world, still have a few database scripts that must be run manually during a production deploy (don’t ask me why). Or pick your poison in the real world you are contending with, often infrastructure and platform tasks are the last to be fully automated.

So during a release, engineers need to be on hand and ready to go when called upon. If however, we now all subscribe to a tooling solution that acts as a common backbone for our release orchestration … the engineers know exactly when to do something, if they can do it early, and when they can go home afterwards. There is the value proposition for the engineer, less time spent doing tedious work during a weekend, when they would otherwise prefer to be at home.

The value proposition to the release manager is real-time insights into all this activity during a release. The automated tasks flow through just as you would expect them to, with up-to-the-minute status of their progress until completion is reached. The remaining manual tasks provide as-good-as-it-gets status updates, also in real time, and serve as the next target of opportunity for automation prior to future release(s).

Changing Human Thinking About RO Perspective

One thing that always seems to be missing from a number of vendor’s offerings I have reviewed in the release orchestration marketplace, is a view from human centricity. Most release orchestration tooling is obsessed with task management, integrating it with calendar scheduling and designing it for maximum efficiency in getting the next step done. If however, our vendor community would embrace the idea of intelligent calendaring a bit more (a future article will discuss this characteristic more in detail), the offerings could form a more human-centric view of the release orchestration process.

Imagine what it would be like for either an engineer or a change manager to log in to the release orchestration toolset and have a personalized dashboard outlining the tasks that need to be approved, worked on or completed to support the all pending releases for that user (based on integrated workflow and intelligent scheduling). An automated red-yellow-green attention draw, for work that may be behind, on schedule or completed early, would greatly improve the human experience of release orchestration. And an integrated departmental or team-focused view of the work effort from manager or employee-centric views would add to job performance critiques—proof of superior, or inferior performances over time.

This represents only a simple recognition that humans should not be stuck in the machine of release orchestration tooling, slave-to-a-machine-centric view of actions to be completed. Instead, a vendor could recalibrate a more human perspective—a subtle alteration in perspective adds an exponential lift to the value of this release orchestration toolset, and it seems to me, any vendor could be there first … if they wanted to.

To continue the conversation, feel free to contact me …

May 4, 2016
The Nirvana of Gated Progression

A highly functioning enterprise-class DevOps set of services has a fundamental goal of full automation. To be honest, having the right DevOps strategist (or visionary) leading this effort is as important to achieving this goal as the combination of all the challenges and topics that must be addressed. So for the sake of argument, will we assume you have one (if not, feel free to contact me—I am looking at the moment 🙂 ).

With key leadership in place, the first target is always the automation of the build and deploy process. This could easily be considered the core of DevOps or its heart. But once this automation has been achieved, the trickier topic of release orchestration begins to emerge. Strategy is to release orchestration what taste is to food: Having the right one does make all the difference.

To progress our code to production, the inherent assumption is that all testing has been completed successfully. Sounds like common sense. But then, how much of that testing occurs with human hands on it? The user acceptance testing (UAT) discipline inherently requires a human at the wheel. But the reality is, for every time a human must interact with the progression of code from the development-class environment through all the testing environments (integration, performance, user acceptance and QA) on its way to production, the momentum of innovation is slowed down—all the way down to human levels.

The Complexity of Testing Types

The first automated testing begins to occur in development or (DEV) class environments, where the programmer is under strict instructions to implement test-driven development. This requires new feature/functional testing as part of the build or deployment automation into DEV. Either the build or the deployment cannot be said to be successful “if” the accompanying tests are not “passed.” This also represents our first “gate.” Sophisticated release orchestration (RO) software can examine “flags” set during the build or deployment process to see if the testing has been completed successfully. If so, the code can progress to the next testing level environment within the software development life cycle (SDLC).

Another useful type of testing completed early in the SDLC (most often in DEV) is the examination of code coverage. Development organizations can begin to examine how much of their code is put to the test, and set arbitrary percentages as a minimum standard before the target application is able to progress. The addition of alternate types of testing occurring in DEV class environments does add to the quality of the innovation effort. However, it brings with it an increased labor effort to automate both progression—and audit—of the version under test.

For instance, feature/function testing may use a single repository to store the results for a given version of the code. I go to my testing repository and I can look up the results from version 1.4 of my application, or version 1.5 of my app, as many as I desire. Should a regulatory official or internal auditor ever ask me to prove the quality of version 1.5 of my app, I need only consult my testing repository and show them how many tests were executed, and how many passed or failed before this version of my app was ever deployed into production.

When you add code coverage testing to the process, I now need to consult two different testing repositories, producing two separate reports to assess the quality of this same version of my app. There is nothing wrong with this, but it does create more work or effort to produce than it would if a single testing repository was used. The problem only increases as we keep increasing the types of tools we use to prove quality. Security testing, for example, (ensuring code does not have inherent security holes) tends to use different tools, or, worse, processes that require human peer observation, which again can add effort to create the automation and the traceability for audit. These are critical types of testing (and/or processes) so they must be done, but they also come with an inherent cost in automation effort.

Small shops may not worry about these kinds of concerns, but large organizations will, particularly if they are still in the process of transitioning from legacy waterfall methods (where controls were high) to new agile methods (where it feels to auditors like the wild, wild west). Compiling a test log that matches the version of code under scrutiny such that it can be reproduced automatically will restore the faith of both internal auditors and regulatory officials who come asking. But the addition of each new type of testing tool will require more effort to maintain the ability to compile those test logs.

The Complexity of Testing Responses

Next, consider the impact of non-binary responses to automation efforts for gated code progression. Does my feature work (yes/no)? This is a binary evaluation, a true/false or yes/no that is easy to set my progression flag for testing response. It is another matter to evaluate the results of my code coverage and ensure this app is above 70 percent or between 70 percent and 90 percent. These kinds of questions are not binary (in other words, there is a real result to compare to, not just a pass/fail), and require further effort to set a binary flag for the gated pass/fail. When the evaluation question does not intend a binary answer at all, such as a grading of 1 to 5 where 1 equals horrible and 5 equals outstanding, I am now most often forcing a human to make an intelligent decision as to whether this code should progress.

Multiple testing repositories require additional automation effort. Multiple kinds of responses require additional automation effort, and the standards by which the gates pass or fail also may change over time, requiring tracking of the goals at the time the tests were taken. Creating independent test automation that can be called from the build process or the deployment process is wise. Associating this test automation (whether scripts or simulations, etc.) with the version of the application code is wise. And working to reduce the number of repositories to examine is wise. Having de-coupled test automation strictly from a build allows it to be re-executed in more advanced testing environments when the need arises.

Charting the Course to Nirvana

RO benefits from as full a level of automation as possible. I will discuss the RO topic more fully in another article, but the foundation of automated change progression relies on a gated history of proving quality. As long as the code under review continues to pass testing, setting progression flags to green or yes or pass, the RO software can continue to progress it along its way to production. Failing code must be sent back to the DEV environments where development teams can address the issues before making its way back through the automated process once again from beginning to end.

If the organization has made the intentional decision to keep humans substantially involved in the testing and evaluation cycle, it must realize a few impacts of that decision. Daily movement to production becomes less likely—most often it takes a week or two for changes to undergo significant human scrutiny (depending on the level of testing automation). The results of human intelligent decisions also must be chronicled in the testing repository to preserve the history for audit. This can lead to a “blame game” kind of thinking when something goes wrong, as it is immediately attached to “who” approved it, instead of to “why” they did so. And obviously the cost of labor is always the highest cost element in any innovation cycle, so buyer beware in this scenario.

On the other hand, “Nirvana” is the state when all testing (except UAT) is fully automated and gated, and provides the foundation for RO software to move code into production as often as it can prove quality. This could mean fully automated releases instead of release weekends. This could mean multiple releases per day into production. This could mean that releases themselves are no longer considered “events,” but by default are now “non-events.” Nirvana does not endanger the lives of change and release managers—it actually gives them one. For a “change” (pardon the pun), they get to discover something they vaguely remember … a life outside of work.

To continue the conversation, feel free to contact me.

April 26, 2016
The Transience or Permanence of Provisioning

I realize the title of this article may conjure up ideas about the impacts of entanglement from quantum computing, but no, we are not there yet. Instead, I am referring to the benefits an organization can achieve from determining which environments in its software development life cycle (SDLC) should remain permanent and which are better candidates to be transient to understand the discreet costs of innovation (ideally if virtualized).

First, to keep an even playing field, make sure your DevOps services are using the “build once, deploy many” philosophy. Under this thinking there is a single build constructed that is deployed unilaterally into each class of SDLC environment (not a unique build for each class). Uniformity under test eliminates variability of results. Uniformity of the basic process of deployment, builds consistency and predictability also will reduce errors and the time it takes to resolve them.

Second, the actual computing capacity you deliver any environment on may be virtualized, whether you provide your own bare metal assets (private cloud foundation) or if you use a public cloud entirely is less the issue. Understanding whether a class of environment is transient or permanent only affects how you plan for the costs and policies of operating it, and it leads back to the discreet cost of innovation as a result.

The Cost of Creation

When this topic is usually considered the first and immediate point of view is usually R&D, or the development class environment. And yes, the ability to provision a DEV class environment and then tear it down when the effort has completed is the normal use-case to justify virtual infrastructure at all (public, private or hybrid cloud usage). The obvious benefits include having no “throttle” on how many concurrent efforts might be spun up.

Transient DEV environments also build in a concept of failure, or elongated “playing,” not intended to produce a specific result but intended to dabble with innovation to see what “might” be possible. All of this without the capital investment it would take to build and maintain an R&D lab to accommodate it, or set aside fixed computing capacity for it. DEV efforts with intent (i.e., part of a formal project) can be tracked for costs discreetly with that planned innovation. DEV efforts without intent (i.e., more like what used to occur in R&D labs) can be associated to the overhead costs of all innovation.

The Cost of Evaluation

But beyond DEV, the next major class of environment to consider is the integration (INT) class. If your company has more than one application, has apps that work together or has applications with upstream or downstream dependencies, you likely need an INT class environment. The major responsibility of this class environment is to provide “something to point to.” Your app (undergoing change) has an environment where it can point to all the other applications to test how the interconnections work. Logically, this class of environment is a better candidate for permanence. This is because you can rarely predict which apps may need this service at which time across the entire portfolio. Many apps undergoing change may be using this at one time to insure they all work with each other (i.e., assessing their interoperability with future versions, early, or prior to production).

Another reason for maintaining a permanent INT class environment may involve your interaction as a vendor to other companies who “point” their apps at yours as a service provider. Even organizations who have little interaction between their own apps, may well be a significant provider to a wide swath of customer applications. If your customers need a place to point to, the INT class environment is typically the one you want. So in this instance permanence is the logical need.

Next up, becomes the performance test (PT) class of environments. Emulating production, or being able to project differences between the platform you use, and what production provides is key to success. However, this class of environment is a perfect candidate for transience (especially if your development and production environments are already virtual). Why maintain expensive hardware on the floor, only to conduct periodic testing, which requires massive reconfiguration and preplanning. Dump it. Instead, use a virtualized platform where provisioning is near instant and can exactly mimic both development and production – remember: build once, deploy many.

Another significant class of environment in the SDLC to examine is the User Acceptance Testing (UAT). Logically, this class environment may get a lot of usage, but ideally only after the app has passed testing in development, INT and PT. Because of this, the UAT should be transient as well. A good candidate to tear down when not in use. You may wish to plan for a setup and tear down of this class to mimic what happens in formal development efforts.

Finally, the quality assurance (QA) or preproduction (PreProd) class environments, if in use, are designed to emulate the current state of production completely, allowing the app-under-change to run against a more sophisticated type of INT class environment (with possibly copied production data, etc.). This happens more often when the earlier INT class environment contains a great number of other applications under change. This one provide a “cleaner” place to point to, only production operating apps. The QA environment may also be used as an after action, when an error is discovered in production for debugging or more extensive monitoring. Obviously, if your organization finds a need for this class of environment, it will carry the permanence characteristic.

The Cost of Operations

Production is by nature permanent. Virtualizing production offers the ability to scale on demand. But a scalable production solution benefits from more mature DevOps services (whether you own the data center or not). Consider for a moment, the number of errors attributed to code post a DevOps embrace. The number of errors goes down, the time to resolve them goes down, the size of the effort gets smaller for less risk, and higher speed. This combination means production is going to suffer far less from aberrant code running amok (asking for tons of memory, CPU or disk due to error). Those kinds of problems get detected earlier (because of DevOps) in the SDLC process and hardly ever reach production after DevOps is engaged.

With a mature use of DevOps, it means that any kind of scalable solution of capacity-on-demand will get engaged for only two reasons: a dramatic increase from customer demand (the ideal solution) or a denial of service attack attempting to drive your costs higher without real customer use. The monitoring solution you employ will help decipher the difference. If your application has logging features built into it (i.e., you produce a log detailing what customer is performing what key feature in your app and when) you should be able to avoid the latter scenario with an automated monitoring detection effort.

New Abilities to Measure the Cost of Innovation

Understanding the cost delta of transient and permanent environments is worth examining. It will help you understand better the cost of innovation on an annualized basis. And it may also reveal how much real effort goes into testing of each kind of class. If user adoption is suffering for example, and you determine that UAT class environments are largely unused or torn down, you know better where to address the problems earlier in the cycle. The same can be said for PT as well, how much test, for what kind of scaled performance in production.

If your organization maintains INT or QA class environments you may want to allocate their costs to SDLC overhead as they tend to be permanent, and value shared. The development, PT, and UAT environments, if transient, represent the variable cost of a particular innovation effort and can be tracked with good discretion. When considering the timing, value, and benefits of virtualization, you may wish to factor in where you are on the DevOps continuum. If you have had DevOps running for a while, the timing for virtualization and scalable capacity on demand may be better, than if you are just starting out.

Keep in mind that most organizations who have transitioned away from waterfall and into agile, will lose a good deal of the former financial controls that worked in waterfall, and do not work as well in agile. DevOps delivery is also capable of moving change at orders of magnitude higher speeds than former manual methods. This combination makes trying to cost innovation discreetly more difficult. Being able to associated transient environment costs with formal development efforts (ones with intent) may restore some of that financial discretion. It may also help identify the true overhead costs of innovation located in the operation of environments classed better as permanent.

To continue the conversation, feel free to contact me.

April 19, 2016
DevOps’ Cultural Myths and Logical Fallacies

Remember that guy you knew in college who was working on his fifth or sixth degree with no plans to stop? At some point the idea of real work must have terrified him, and instead what appealed to him was a continual state of learning. If you have the funding to master five or six disciplines, and perhaps not the ambition to get going in any one of them, why not stay and explore the next one?

But if you work in IT, chances are you did not stay in college for more than a few areas of study, and since you entered the work force, your future options dwindled down to two: become a specialist (deep thinking in a singular discipline, but not too broad) or become a generalist (broad thinking across multiple disciplines, not too deep in any one of them).

Managers tend to start their careers as specialists who do so well they are promoted to manager, where their career shifts and they become generalists. From time to time they dabble in a particular area of study to speak the lingo, use the right buzz words and understand what is coming, but the role they play is inherently generalist, as they are forced to deal with a number of topics beyond the realm of any one particular area of study.

We tend to associate the title “architect” as being the specialist who is deep in an area we are forced to contend with. Examples might include, “We need to hire a network architect, or a database architect, or a software architect …” the list continues. When we do this, we have certain expectations. We are looking for the “guru” who can solve our network problems or database problems or security problems—wherever the skill set was that caused us to hire this specialist. If our new employee cannot solve our problems, we begin the search for another who can solve them.

But the candidate has expectations as well: The network guy may have a little knowledge about databases but does not expect to have to perform at the level of his peers in that discipline. If we hire a network guy and then ask him to do everything in every other discipline, we are the ones out of line and he will likely flee our employment in haste, calling us crazy on the way out.

The Cultural ‘Frankenstein’

To create a true master of the multiple disciplines that exist within IT would require creating a modern Frankenstein in networks, servers, operating systems, storage, database, middleware, software and mobility. And, of course, this list is not limited to one expression of each type, but indeed multiple expressions of each type. So for example, our database discipline likely includes deep expertise in Oracle, DB2, SQL Server, NoSQL, MySQL … you name it. If it is popular or relevant to our current stack, our Frankenstein must know that one.

As DevOps has gained in popularity, we have conflated certain aspects of it and driven ourselves off the highway of logic. Too often, proponents have taken the concept of co-location from the agile beginnings of DevOps and applied it to the operational endings of DevOps. Taking down silos is mistakenly applied to co-locating IT operational staff. This conflation occurs billed as the panacea it once was to co-locating business staff, with the developers producing products for them. In the strictly agile context, this works well. Applying it to IT personnel on the back half of delivery is the least effective means of creating Frankensteins. Co-location is a good way to share ideas, from user back and forth to innovator. In general, sharing ideas between a network guy (with limited interests outside his discipline) and middleware guy (who could care less how the network is configured in detail) has limited success.

These operational IT disciplines do interact, but only as they have need, which is significantly less than having to sit in the cube 4 feet away from a counterpart involved in a differing discipline. Co-location is not the cure for breaking down silos or increasing communication—common backbones are. Imagine the example of the medical profession: A hospital provides the common backbone for the neurosurgeon, the orthopedic surgeon and the heart surgeon. These disciplines are inherently separate, and if you have a particular illness, you are going to want the one who specializes in the thing you need done. While each doctor understands general health aspects, they specialize for a reason—a reason you want and need—when it is your time to seek help.

The hospital provides the common facility, the common support staff, the common electronic systems and tools we use. Our medical records are stored in one place, no matter what type of treatment we seek, so each specialist has to enter them there and can read them there. But distinctions by practice remain. There is a special surgical floor for heart surgeries, where they do not conduct orthopedic work or neurosurgical work. The tools may be slightly different; the medical journals are. The training is different. The experience is different. And none of these skills transfer by osmosis of co-location if you put all three doctors in one central cubical cluster. It takes years to get where they are. The same is true of our IT specialists.

If instead of co-location we looked more at common systems for moving, documenting and articulating change. If we gave limited access to our DevOps tooling to the specialists, confined to the niche they have expertise in, then instead of co-location we could have remote-work-concurrence. Experts from all over the country could work on the movement of change without ever physically seeing each other, but with full visibility into what is happening in real time through a common lens. Network operations center (NOC) personnel remain physically in the NOC, but can see the entire stack from that location and change only their portion as changes are required from there.

It is the common backbone of DevOps systems that breaks down the silos of change transparency and change participation. Prior to DevOps we were all using different tools to move change and track change. With DevOps we can begin to articulate change and document it, in the same tool or interface. Meaningful access is granted only to the portions of tools and only where it is needed. Database guys have full access to that platform and nobody else does (other than to see it). The same model works for every other specialty. And before you know it, you have a concert—or rather a symphony of moving parts—instead of disjointed sounds no one can comprehend.

Organizations that hold to the idea that staff will become Frankensteins, equipped with deep skills in all areas because they share cubicle clusters, would do better buying lotto tickets. Where the culture will require attention is in demonstrating to the impacted teams why participating in the common backbone approach has merit for each of them. For engineers who have never had to use “something different” before, this takes work, but is the real win.

When complete, what has truly happened is a conversion to a culture of unity. The work of the genius is still protected in this method; in fact, it is amplified as now it is visible to the entire organization. So even the prima donnas should be happy with the approach. And the ability to orchestrate change never will have been so high, with so much real-time interaction, as after the embrace of a common backbone of DevOps systems. This should offer executives new insights into product creation and evolution. In this sense, the cultural changes reach from top to bottom of the organization, and they are achievable.

To continue the conversation, feel free to contact me.

April 6, 2016
Funding Failure by Intent

Why is it that when we purchase a lotto ticket, every part of our brain tells us we won’t win, yet some portion of it is able to inspire hope where there shouldn’t be any? Could it be the same part of our brain that inspires hope in the face of nearly certain failure has been nurtured by the software development projects we have been exposed to over the years?

It would appear that the ability to prognosticate has been the foundation of nearly every waterfall project that had a timeline of three years or longer. It is this myth of “certainty” that agile methods are designed to eliminate. In our new approach, we simply break down and digest the scope of work until it is small enough for our risk tolerance, and easy enough to implement in our abbreviated time horizons. We try, we evaluate and we try again.

But no matter how low we believe the risk to be (and generally the reward we expect from it), risk does remain. And now instead of a three-year project plan well-equipped with milestones, corrective actions and projected costs, agile gives us only the current sprint, based loosely on our historical velocity and a literal “addiction” to continue this process going forward without any real regard for a predictable end-point or total project cost. Continuous delivery infers continuous innovation (emphasis on the word “continuous”). And if done right, why would a business ever choose to end innovation that works?

Imagine the analogy of Mozart sitting down to compose a quick advertising ditty that your company could play as background music on its website. Mozart begins with a few bars, and notices a theme that is worthy of far more exploration … so he does. He continues exploring this until an epic symphony emerges. This was not the plan and does not meet the original requirements. This could be considered scope-creep in the waterfall world, and thus a failure, as no one stares blankly at a website for the 30 minutes it takes to hear an entire Mozart epic … or do they?

Every audience grasps the concept of background music we pay little attention to but subconsciously understand it improves our experience. But nearly every audience also realizes the beauty of a Mozart epic, and would gladly buy CDs, MP3s or concert tickets to hear one of those over and over again, relishing every note. The question this analogy begs is simply this: Would Mozart’s employer—more specifically, his development manager—have been free to fund this “failure” or not? What happens when a project changes direction significantly away from the first vision we had of it? Does DevOps enable us to pursue this radical change in direction, or are our legacy financial systems and methods binding us to a waterfall delivery and reporting structure—and, in so doing, limiting what we will ever be able to accomplish through radical innovation?

The beauty of agile construction and DevOps delivery is that we may deliver “unexpected” innovation. Nearly every time a user “sees” what we have delivered to meet their need, they immediately think of “x” more things we could do to make it even better. This may result in taking a project in a radically different direction within the confines of a single financial year. To further enable our ability to do this, our legacy financial funding and reporting systems must evolve to match this demand. Our new risk profile must include the ability to fund failure on purpose, understanding that failure itself is part of the maturation process of innovation, not the death-knell of the company.

A more honest approach to funding agile/DevOps efforts would be to determine the annual investment the organization wants to make in certain capabilities, and innovations it expects to see in each thematic area. Overarching themes are funded on an annualized basis; however, great autonomy is granted and expected within the themes to ensure delivery is achieved. Pushing financial responsibility down in the organization to as low a level as possible has the effect of pairing the team leaders with the costs of what they do. Spending too much time on delivery of any given feature or function may cause the team to re-evaluate the approach and propose a better solution. This is more unlikely when budget responsibility is managed several levels above the team doing the work—in that instance, the budget resembles more of an infinite resource they need not be concerned with, until it is too late.

When proponents of DevOps claim it is changing the culture of a company, they are not limiting this view only to IT. Tertiary services in a company, such as the supporting financial budgeting systems, hiring systems and even contract management systems, may require significant change to match the pace of delivery and innovation DevOps delivers. As an enterprise looks to implement DevOps, it should look broadly at what else may be required to fully enable the pace of innovation it is capable of.

To continue the conversation, feel free to contact me.

March 23, 2016
Continuous Improvement: The Key DevOps Distinguisher
DevOps has become a well-accepted and growing component for delivery success, shown in the State of DevOps Report among other respected publications. But even as it becomes more ubiquitous, it’s naive to think every organization will cease having delivery concerns. Implementations will differ widely, and adherence is essentially self-reported. So what is it, then, that separates those organizations that struggle with effective software delivery for years from those that consistently shoot further past their competitors?

The fuzzier nature of the culture and relationships aspects of DevOps obstruct the potential for an ISO- or CMMI-style certification—which is likely a good thing, though it also opens the door for DevOps to be simply a re-branding exercise without any teeth. As a guideline, the CAMS model is still an excellent overview of key DevOps principles. It lays the groundwork for measurement, feedback and sharing that set the stage for improvement, yet stops before making continuous improvement a top-level concern.

The rate of continuous improvement is the metaphorical equivalent of compounding interest. It takes just a simple illustration to show how starting further behind, but focusing on continuous improvement, can take you past those that have more initial resources.

Prioritizing DevOps for Continuous Improvement

DevOps with a focus on Kaizen or continuous improvement can catapult a company’s ability to effectively deliver software. Unfortunately, it falls squarely in the Important/Not Urgent quadrant in Stephen Covey’s model, where good ideas and good intentions are continuously frozen. An inability to keep momentum and traction in this state affects many companies.

To reach our goal of continuous improvement in the DevOps implementation in a company, first we must examine how to successfully drive internal initiatives in general. As a default behavior, companies commonly relegate themselves to just making it through quarter to quarter, delivering mostly incremental customer features. More ambitious initiatives draw lots of excited talk in executive suites, yet often stall and get delayed in actual implementation. Why is that an all-too-common occurrence?

There are three main hurdles:
- Inconsistency of shared goals and priorities across all levels of the organization: There’s often a gap somewhere around either middle management or line management where initiatives can tend to slip. It doesn’t hurt to over-communicate and ensure it goes across all levels, especially when important changes are occurring.
- Lack of individual accountability and empowerment: If communication is adequate, do individuals know not just the goals but also what they can and should be doing to reach them? This aspect is in principle tackled by official systems such as management by objectives (MBO) or objectives and key results (OKR). You can find success or failure in many systems, so rather than debating the merits of various people management systems, make sure to focus on the key goals and actions chosen.
- Do people actually have the budget of time and potentially funding to take action? Without time for action, there is only a nebulous intention—and perhaps a slight feeling of guilt at not having made progress.
Simply identifying the above obstacles and addressing their existence can be eye-opening and likely will cause common-sense measures to emerge to align the organization. After the initial quick wins, there certainly will be more difficult steps to follow that are specific to each organization. The end result is certainly worthwhile.

Unblocking the organizational obstacles to DevOps implementation, not as a one-time project but an evolving program of continuous improvement, blazes the path for long-term success.

As a further thought, imagine you also accelerate the rate of continuous improvement so it’s not a constant. An amazing example of the power of acceleration is the notion of a solar sail: Accelerating its velocity just 1 millimeter per second seems imperceptible, yet after one day, it’s moving 195 mph, and after 12 days, it’s up to 2,300 mph. “Astronomical” is an apt description of the delivery speed of some famous tech startups. That’s the power of continuous improvement!
March 22, 2016
Changing the Culture of Metrics Analysis

You could say the development and operations teams gave birth to the idea of DevOps, as it is believed the chief benefits are garnered in those two functional disciplines. Because of this, metrics nearly always are conducted from the points of view of cost reduction, speed and efficiency. This makes perfect sense when business treats IT (and IT sees itself) as a service- or cost-based organization. But what if DevOps has a few more characteristics that could actually change that culture? What if, for the first time, DevOps could enable IT to become a real partner with the business organization it serves?

To make this happen, business and IT must begin to share attainable goals. Imagine the impact of a highly functioning, enterprise-class DevOps service within IT; upon the time to market of products and services the business is looking to launch. A proven, consistent ability of IT to deliver on time or ahead of schedule (i.e., reliability) then begins to change the dynamics of what marketing proposes and delivers. Time then becomes a valid metric to consider when looking at customer adoption rates. DevOps usually does not directly impact initial market demand, but the impact of delivering new services or features months or years ahead of previous projections can dramatically impact time to market. For example, if your business is able to change its billing model or billing methods six months to a year ahead of every one of your competitors, what impact might that have on market adoption? At that point DevOps becomes a key business differentiator, until the rest of the universe has it.

Consider an additional impact of a highly functioning DevOps service on feature delivery or the feature road maps themselves. The implementation of DevOps inherently delivers smaller units of change on a higher frequency, reducing risk and allowing iteration and alteration of future plans based on the real-time feedback of change success. Marketing focus groups may provide an original road map, but the impact of DevOps can be integrated back into those same focus groups. Instead of imagining what might be possible, DevOps allows delivery of what is possible to see, feel, touch and prompt further imagination. The net impact is an increase to the pace and depth of innovation itself. You know what your customers really want, and you can give it to them quicker. What kind of impact does that have on the reputation of your business if it becomes a market leader because of this? Or what happens when a large organization is then able to deliver with the speed, agility and flexibility of a small firm?

Cost Reduction vs. Revenue Generation

Now consider the value of cost-reduction metrics compared with revenue-generation metrics. It is easy enough to determine that the pre-DevOps delivery cycle might have been 12 weeks per feature as an example, or that each build took six hours to complete and now post-DevOps takes only 12 minutes to run. Metrics like these will begin to show an increase in productivity by existing development staff—or, maintaining the same feature delivery rate, a reduction in the number of personnel needed to achieve them. Similarly, it is easy to show a reduced number of trouble tickets post-DevOps, and a concurrent reduction in the average time to resolve each ticket. DevOps then has enabled the CIO to continue to take arbitrary budget cuts demanded by the business each year. This is not partnership—over time this is a recipe for the reduction, pace and quality of innovation (the smartest people leave first during layoffs; they don’t need to stay).

However, even significant cost-reduction metrics pale in comparison to revenue-generation metrics. As an alternative, imagine what the net growth in customers of a given product might be, over a six-month period of time, because a desired feature set was delivered early? And what if it is two to three years before your competitors can implement the same high-functioning DevOps services you already have in place today? If a business makes $2 per-widget profit and captures 20 percent additional market share over six months, equating to “x” million new customers … well, you begin to get the idea. All of the sudden IT is no longer viewed as a cost center by business; it is viewed as a revenue-enabler. The immediate business response is, “If I allocated more budget into the types of staff facilitating DevOps services (running the gamut from coding to operations to marketing), could I gain even faster delivery rates or deeper feature sets or more market share by pioneering what has never been pioneered?”

The two main reasons this realization hardly ever occurs in business today are that 1.) IT has failed at delivering on promises so often it has trained the business to believe it “cannot” deliver on time or on budget; and 2.) few businesses conduct accurate time-to-market metrics measuring the delta of what additional “time” does to the equation, largely because they never had to. DevOps provides the catalyst to begin to re-examine both of these phenomenon.

But it goes deeper than that. Changing the culture of innovation, in which I am eager to test new ideas faster, means I need more ideas all the time. The competitive advantage our nation, our businesses and our careers have always benefited from is our ability to create and innovate. Other nations around the world, or other organizations your company competes with, may be able to do repetitive tasks faster or more efficiently. But rarely do worker-drones create or produce a wealth of ideas. But if we begin to focus on the long-term impact of fostering innovation in our own country, and in our companies, we will remain perpetually out front. The goal of DevOps is not merely efficiency, it is also to spark the imagination. When our focus and our metrics reflect this, we all win.

If you would like to continue a conversation on how to create and implement a highly functioning enterprise class DevOps service, feel free to contact me.

March 14, 2016
In DevOps, Standardize to Minimize Architectural Debt

I recently reading a great Q&A with the Etsy CTO John Allspaw over on The New Stack, which contained some nuggets of good advice that increasingly are important as we continue the journey toward DevOps adoption. Of particular note were statements on the introduction of “new” things to solve problems. In a nutshell, Allspaw reinforced the importance of standardization as a means to reduce architectural debt.

If you aren’t familiar with the concept of architectural debt, it’s similar to technical debt, in which choices made today incur longer term investments of time and money when future changes are introduced. If you settle on a caching solution such as Redis, for example, there are technical and architectural implications. You can’t simply switch out Redis for memcached later without significant changes to both code and the infrastructure.

Even your choice of load balancer today implies architectural debt because of the growing inclusion of scalability as part of the application infrastructure itself. The result is integration with automation and orchestration systems that are not easily changed later on if you decide you want a different solution. The style of service registry you choose is an architectural decision that impacts clients, servers and systems across the microservices architecture. Switching from client-side to server-side will later be costly.

That’s architectural debt.

Allspaw noted the need to not reinvent the wheel when possible, and instead encourage the collaborative aspects of DevOps as a means to avoid reinventing the wheel and potentially increasing the architectural debt load on the organization.

This is not a new concept. There’s a reason enterprises in particular proudly proclaim themselves “Java shops” or “Microsoft shops” and primarily require development in one or perhaps two languages, but no more. Maintainability is a component of technical debt, just as middleware decisions are a component of architectural debt.

So as we continue to see wider DevOps adoption, it’s important that we carefully consider the impact of our choices in terms of the automation, orchestration and integration required to implement continuous delivery and deployment. Once you’ve invested time and energy (and probably budget, too) in Puppet or Chef, VMware or Cisco, OpenStack or SaltStack, you’re going to find an unavoidable amount of technical and architectural debt in development as well as production across infrastructure and networking teams.

Each of these systems, solutions and frameworks comes with its own components. It isn’t just a language, it’s a comprehensive system with management and repositories and agents and reporting components. All of these impose certain architectural requirements on organizations, from needing certain software running to prescriptive details on networking that must be enforced, lest the system break down.

We’re already starting to see that fragmentation in enterprises that are engaged in automation and orchestration efforts. When asked the number of automation frameworks (Cisco, VMware, Python, OpenStack, Chef, Puppet, etc…) in use, one-third today reported using three or more.

That’s not necessarily “bad,” as it could be indicative of standardization across silos (networking wants to use Cisco, infrastructure is already on VMware, and Python is being used to cobble it together by ops), but it also could be an early sign of the kind of fragmentation that causes exponential growth in technical and architectural debt.

Certainly there is room for experimentation in these early days, but it behooves us to maintain a watchful eye on the use of frameworks and systems and languages we employ for automation and orchestration to ensure we don’t end up buried in technical and architectural debt.

Standardization, even loosely enforced, can help minimize the impact of that debt before it spirals out of control.

March 14, 2016