devopsy.kinsta.cloud

Author: Kristian Nelson

Changing Compensation Before it Kills Success

Have you ever seen the sign, “Will work for food”? If you deconstruct the premise a bit, we all work for the rewards systems that interests us. We “trade” our labor (ideally, focus our passion), for a set of acknowledgements that range from survival to pride. And ultimately it is our employer (whether the boss or the board of directors) who decides what our rewards systems will be. That is the common sense part. But when something such as DevOps enters the picture, it alters traditional thinking throughout an organization, and the ripple effects should (and eventually will) reach the HR department.

The most immediate effect of DevOps on HR occurs around talent acquisition. Finding a qualified DevOps engineer that it is possible to afford—and the company can retain—becomes the unicorn of recruiting departments from one coast to the other. The alternative is creating or adopting a training curriculum that can move a traditional programmer or engineer into a fully qualified DevOps engineer. Of course, advice varies on exactly what that is, and if it’s successful, your organization becomes a feeding ground for every other competitor to poach your folks right after training.

Because this HR acquisition problem is so great for most organizations, they can overlook some other underlying issues that continue to occur. These problems have always been there, but DevOps brings a spotlight to them that did not exist prior to its implementation. As stated earlier, we “do” what we are “paid” to do. In organizations that innovate in the software discipline, we have always “paid” that entire division to innovate. Prior to DevOps, the pace of that innovation, was slow and manageable. Once DevOps begins to take hold, the pace of innovation begins to enter the 400mph superhighway, where traditional control mechanisms no longer work.

In itself, the success of the controls over change—transitioning to a high rate of speed—will be determined by how we implement DevOps. It is the other half of DevOps that needs some focus. While our Dev organization is cranking out change exactly as we have always paid them to do, our Ops organization traditionally has been paid for exactly the opposite characteristic. We pay our Ops organization for stability. We want the trains to run, on time. We do not want a 2 a.m. phone call from an angry CEO—or, much worse, an angry customer—because our operations have “broken.” And, of course, the No. 1 culprit for causing breakage in operations is … [drum roll, please] … “change.”

In the yesteryear of Waterfall construction and manual delivery, our Ops teams had time to digest what change was coming, prepare for it, adjust to it and get it implemented smoothly (or at least pretend to). In the current day of Agile construction and DevOps delivery, I might have moved six new builds into production in the time it has taken you to read this article. Ops has no time anymore to digest and prepare for change, so if the quality of that change is not significantly better than it used to be, the angry-phone-call potential goes up exponentially. And yes, there are technical remedies to ensure quality (I will reserve discussing those for another time).

But the tradition of how we compensate remains in place. We comp Dev to innovate, and we comp Ops to remain stable. This makes not only silo thinking, but also mortal enemies during the adoption of DevOps into an organization. All of a sudden, each views the other as potentially causing harm to their ability to continue earning compensation. And they are right to see it that way, as HR has yet to alter the models to create partnerships instead of combatants.

To fix this, the compensation models must be adjusted to pay Ops teams to get involved in the software development lifecyle (SDLC) process from the upfront. The Ops teams should have extensive visibility into the testing cycle of new changes and how that works, and become the organization always asking Dev how it can “prove” its change works. The Ops teams must become the in-process auditors of the Dev teams. This keeps the Ops teams from being compensated for conditions that only exist at the end of the SDLC process, where they feel “dumped-on” by more rapid change delivery. The new comp model for Ops must focus on the number of changes successfully introduced into the production environment (i.e., the quantity of change that works), not just the stability they traditionally maintain.

The Dev team, on the other hand, must begin to be compensated only for after-action success. Dev teams must look at their code after delivery to production—and only if the code is validated (i.e., it works), is adopted by its intended customers (a completely different angle) and performs within expected timing parameters (i.e., it scales well). This becomes the criterion that is the foundation of the Dev team’s measurement of success. This puts an Ops set of glasses on the programmer instead of making him blind to those business issues. The Ops teams become the key advisors to the Dev teams on how to prove quality. And the joint discussions lead to a more sensible monitoring strategy, which is needed across the board.

There’s Compensation, and Then There’s the Bonus

Changing the point of view for the compensation is only a start. Next up: Begin to alter bonus compensation to a more lean model. Instead of large annual bonuses, where the basis for achievements often are colored by my boss’s mood over the last six weeks, organizations should increase the frequency of bonus payouts to match the duration of epic completions—or, if possible, of sprints. The idea is to tie meaningful bonus payouts to the length of time it takes to produce only successful change. If the change works, the company offers additional bonus comp; if not, then the team must improve to secure the next bonus opportunity.

Tie the bonus duration (whether to epics or sprints) to meaningful business impacts. In this way, each contributing employee understands “why” the work he or she offers has meaning to the company or its customers. This kind of bonus system allows for extreme flexibility in the priority of what teams work on throughout the year. Bosses can change priorities or shift them to match what executives see on the horizon. Employees are not tied to some list of annual work accomplishments that in a DevOps rapid world they had little shot of achieving unless they did them in a week. Instead, they have real-world goals, tied to real world time frames, where employees see the reward as they succeed at the work.

Organizations can add even more bonus comp this way if over-and-above work is needed. And “if” organizations compensate both Ops and Dev only for successful change (which implies quality in the SDLC and stability in Ops) they create a new rewards system that enables adoption rather than repel it. Beyond finding the talent they need, organizations need innovative methods of retaining them. Substantial rewards in real time might just be one way to do that.

Finally, being able to see the business impact of the change the joint effort of Dev and Ops teams produce allows for better ability to offer “appreciation” ideas as well. A simple “thank you” goes a long way in employee satisfaction, and when the executive and the team member understand “why” they are being appreciated, they go even longer. DevOps is not just a catalyst reserved to the “techies” who practice it; DevOps is instead a new way of thinking about how a company innovates. This kind of infectious thinking can easily spread to tertiary support organizations such as HR and Legal, if you let it.

To continue the conversation, feel free to contact me.

April 12, 2016
DevOps’ Cultural Myths and Logical Fallacies

Remember that guy you knew in college who was working on his fifth or sixth degree with no plans to stop? At some point the idea of real work must have terrified him, and instead what appealed to him was a continual state of learning. If you have the funding to master five or six disciplines, and perhaps not the ambition to get going in any one of them, why not stay and explore the next one?

But if you work in IT, chances are you did not stay in college for more than a few areas of study, and since you entered the work force, your future options dwindled down to two: become a specialist (deep thinking in a singular discipline, but not too broad) or become a generalist (broad thinking across multiple disciplines, not too deep in any one of them).

Managers tend to start their careers as specialists who do so well they are promoted to manager, where their career shifts and they become generalists. From time to time they dabble in a particular area of study to speak the lingo, use the right buzz words and understand what is coming, but the role they play is inherently generalist, as they are forced to deal with a number of topics beyond the realm of any one particular area of study.

We tend to associate the title “architect” as being the specialist who is deep in an area we are forced to contend with. Examples might include, “We need to hire a network architect, or a database architect, or a software architect …” the list continues. When we do this, we have certain expectations. We are looking for the “guru” who can solve our network problems or database problems or security problems—wherever the skill set was that caused us to hire this specialist. If our new employee cannot solve our problems, we begin the search for another who can solve them.

But the candidate has expectations as well: The network guy may have a little knowledge about databases but does not expect to have to perform at the level of his peers in that discipline. If we hire a network guy and then ask him to do everything in every other discipline, we are the ones out of line and he will likely flee our employment in haste, calling us crazy on the way out.

The Cultural ‘Frankenstein’

To create a true master of the multiple disciplines that exist within IT would require creating a modern Frankenstein in networks, servers, operating systems, storage, database, middleware, software and mobility. And, of course, this list is not limited to one expression of each type, but indeed multiple expressions of each type. So for example, our database discipline likely includes deep expertise in Oracle, DB2, SQL Server, NoSQL, MySQL … you name it. If it is popular or relevant to our current stack, our Frankenstein must know that one.

As DevOps has gained in popularity, we have conflated certain aspects of it and driven ourselves off the highway of logic. Too often, proponents have taken the concept of co-location from the agile beginnings of DevOps and applied it to the operational endings of DevOps. Taking down silos is mistakenly applied to co-locating IT operational staff. This conflation occurs billed as the panacea it once was to co-locating business staff, with the developers producing products for them. In the strictly agile context, this works well. Applying it to IT personnel on the back half of delivery is the least effective means of creating Frankensteins. Co-location is a good way to share ideas, from user back and forth to innovator. In general, sharing ideas between a network guy (with limited interests outside his discipline) and middleware guy (who could care less how the network is configured in detail) has limited success.

These operational IT disciplines do interact, but only as they have need, which is significantly less than having to sit in the cube 4 feet away from a counterpart involved in a differing discipline. Co-location is not the cure for breaking down silos or increasing communication—common backbones are. Imagine the example of the medical profession: A hospital provides the common backbone for the neurosurgeon, the orthopedic surgeon and the heart surgeon. These disciplines are inherently separate, and if you have a particular illness, you are going to want the one who specializes in the thing you need done. While each doctor understands general health aspects, they specialize for a reason—a reason you want and need—when it is your time to seek help.

The hospital provides the common facility, the common support staff, the common electronic systems and tools we use. Our medical records are stored in one place, no matter what type of treatment we seek, so each specialist has to enter them there and can read them there. But distinctions by practice remain. There is a special surgical floor for heart surgeries, where they do not conduct orthopedic work or neurosurgical work. The tools may be slightly different; the medical journals are. The training is different. The experience is different. And none of these skills transfer by osmosis of co-location if you put all three doctors in one central cubical cluster. It takes years to get where they are. The same is true of our IT specialists.

If instead of co-location we looked more at common systems for moving, documenting and articulating change. If we gave limited access to our DevOps tooling to the specialists, confined to the niche they have expertise in, then instead of co-location we could have remote-work-concurrence. Experts from all over the country could work on the movement of change without ever physically seeing each other, but with full visibility into what is happening in real time through a common lens. Network operations center (NOC) personnel remain physically in the NOC, but can see the entire stack from that location and change only their portion as changes are required from there.

It is the common backbone of DevOps systems that breaks down the silos of change transparency and change participation. Prior to DevOps we were all using different tools to move change and track change. With DevOps we can begin to articulate change and document it, in the same tool or interface. Meaningful access is granted only to the portions of tools and only where it is needed. Database guys have full access to that platform and nobody else does (other than to see it). The same model works for every other specialty. And before you know it, you have a concert—or rather a symphony of moving parts—instead of disjointed sounds no one can comprehend.

Organizations that hold to the idea that staff will become Frankensteins, equipped with deep skills in all areas because they share cubicle clusters, would do better buying lotto tickets. Where the culture will require attention is in demonstrating to the impacted teams why participating in the common backbone approach has merit for each of them. For engineers who have never had to use “something different” before, this takes work, but is the real win.

When complete, what has truly happened is a conversion to a culture of unity. The work of the genius is still protected in this method; in fact, it is amplified as now it is visible to the entire organization. So even the prima donnas should be happy with the approach. And the ability to orchestrate change never will have been so high, with so much real-time interaction, as after the embrace of a common backbone of DevOps systems. This should offer executives new insights into product creation and evolution. In this sense, the cultural changes reach from top to bottom of the organization, and they are achievable.

To continue the conversation, feel free to contact me.

April 6, 2016
Integrated Testing Without Integrated Migraines

I am always amazed at how many practitioners (or, rather, managers of) tout the benefits of test-driven development (TDD). This inherently changes the role of the traditional developer to one of developer and tester. But it would appear that as long as I am able to construct a test for the innovation I wish to create, all is well in the universe, and I can proclaim the full success of automated test integration with my DevOps processes, sleeping quite well at night from then on.

But in the world before yesterday, testers tend to think differently than developers do. They evolved differently, bearing the scars of applications that tested just fine on the developer’s laptop but somehow died under the myriad circumstances that occurred in the world beyond that original safe harbor. In point of fact, human testers with a variety of automated tools tended to very often “prove” to developers that their sanctified code was not in fact so pious, but actually more “holy” (that is, containing holes) than they thought. Thus, remained the cycle of code/test/code again until the gauntlet to production is navigated successfully.

There is also a certain reality about testing that developers cannot easily overcome. The only category of testing than can easily accompany a new feature or function is primarily functional testing. “Did the function work?” would appear to be the kind of testing a developer could construct to accompany his/her code. Notwithstanding the fox-in-the-hen-house analogy, there are quite a few other kinds of tests that are equally valid in that code moving forward to production. Most of these tests are not easily constructed by the original developers.

One might consider performance testing a bit important. In this instance, I am not only concerned with the validity of my new feature, I am also concerned with its performance (timing) relative to how a user would experience it, and inclusive of the conditions of my production environment. For this to be rightly measured, I may need to ensure all integration testing also has has been completed successfully. And assuming all of this completes as expected, I may still want users to test my app—you know, to see if it is actually what they wanted in reality (after decrypting the requirements from the ancient Egyptian notes of the stories they submitted). Beyond this, some functions appear largely in a batch context, or perhaps only when proximity is achieved in a mobile app, so context is equally important to testing.

The Nuances of Integrated Testing

But getting back to our original question, let us assume I can test something, and let us assume I have tested it. Where do we go from here? Once the validity of my new feature has been tested, should I decouple the test execution from the build automation and move it to a list of regression tests I only execute periodically? That would seem logical, as this test might need to be run again during post-integration testing (particularly if it manipulates data with upstream dependencies). What is the “value” of my test past that first successful occurrence? If it has value, then I must begin to look at all of my testing in the same way I look at code: It needs to be version-controlled.

Pairing functional tests to the versions of the code they validate will allow me to roll test automation forward or backwards to match the version of code in the target testing environment. So, for example, if coding build version 1.5 is in my integration environment and it fails, I can roll it back to version 1.4 and concurrently roll back the tests that were valid for the older version as well. That’s better than using tests designed for version 1.5 of my code, as testing new feature validity against old code is sure to generate errors.

But treating test scripts (or entities) the same way you treat code with version controls is something most organizations have yet to get their heads around. Deploying testing as a part of the post-build deployment automation is equally daunting.

And, of course, none of this addresses the more fundamental question: Does my test have value at all? Because I can test a feature or function, should I test it? Is this feature or function critical to the user experience? Am I testing it in such a way that the user experience will be evaluated? Testing whether a date field allows only valid data to be entered into it seems important the first time we do it. But once the characteristics of a date field have been so ubiquitously defined by software libraries, advanced programming languages or database structures, perhaps continuing to validate that input is excessive and provides diminishing value. Testing the right thing matters as much as testing at all.

Then let us assume we have found the Rosetta stone of items to test and we have created an ability to test this feature with scripts passed down from Olympus. If the continued validity or performance characteristics of this feature remain important to the user experience, perhaps we have not found only something to test but also something to monitor. Here is where we begin to think two years into the future. If part of my DevOps continuum is an ability to integrate version-controlled testing as a capability, why not integrate version-controlled monitoring, as well? (Here’s where the Ops guys of DevOps finally get interested.)

Instead of examining testing from only the perspective of the developer chained to the dictate of TDD, why not examine the life cycle of “evaluation” from a more holistic perspective of coding, integration, performance, users and, ultimately, a monitoring point of view. If my monitoring tools are capable of configuration adjustment, parameter adjustment and ceiling- and basement-level adjustment, then through reading external files or by exploiting API’s shouldn’t we be looking to pair our monitoring profiles with the version of applications we deploy into production discreetly by version? Perhaps a less useful test simply gets retired and a more useful test evolves into a program characteristic that we monitor until the feature itself gets retired.

Integrating the Ops team into discussions about automated monitoring is sure to pique their attention. If nothing else, the ability to turn off monitoring tools during deployments and turn them on again afterward will cut down on “noise,” and help to focus resources only where they are truly needed when alerts emerge. Understanding the difference between how production is monitored vs. the unique needs of the test environments—with the ability to incorporate both into the automation of DevOps—is equally appealing.

In any case, the evaluation of an application includes not only the original tests to ensure validity, but also the monitoring that occurs to ensure continued performance as expected. Thinking about where tests begin and where monitoring is more effective or needed can help to keep both disciplines modest and focused only on what is critical, rather than both attempting to take on the weight of the world. The goal of DevOps is not to increase the frequency and duration of headaches, it is to eliminate them.

To continue the conversation, feel free to contact me.

April 5, 2016
Resurrecting the CMDB Through a Singular Path to Production

Change consistency (or the predictability of how change is deployed) followed by change transparency (or the visibility into the state of change at any stage in the software development life cycle [SDLC]) are two key benefits of embracing a highly functioning enterprise-class DevOps set of services. Most organizations are attempting to transform themselves from a widely varied manual delivery system to an automated, template-based delivery model that DevOps relies upon.

Along this journey a key fact soon emerges: We cannot allow “manual” tweaks to our environments and maintain the predictability, visibility or the stability we want. To be truly consistent, process must be fully automated. A process that relies upon human interaction (whether planned or unplanned), is subject to human variability that can impact its success. In short, we do not want a deployment process that has humans coming along after it, to “clean it up.”

Thus a key principle of DevOps, is that it becomes “the singular path to production.” We want all changes going in to production to have gone through our DevOps systems and processes. Our goal becomes to reduce and eliminate human reliance in every interaction. As an example, we automate test execution, then test evaluation, then test gated change progression. We use “smart” deployment technologies to overwrite what is different in the destination environment, forcing it into compliance with our stated version of changes that “should” be there.

In effect, we “step on” variations, overwriting them every time we initiate a deployment. This quickly teaches our freestyle engineers who make changes on the fly that their work will get “stepped on” with each new deployment (perhaps daily or hourly). Most engineers quickly get the hint that to preserve their work, they need to include it in the deployment process itself (updating the proper configuration file, or adding the right series of steps in the deployment execution automation itself). Otherwise, they will face a daily onslaught of repetitive tasks that inevitably will overwhelm them.

In this sense, a properly implemented DevOps system is “self-cleaning.” Paired with an executive mandate to have all change go into production through DevOps, another benefit emerges that may bring the CIO glee. Just a few years ago, IT organizations were bent on the idea of documenting change (therefore understanding it better) through the means of a configuration management database (CMDB). Those projects tended to be large, costly, and doomed to minimal success. Typically, the CMDB effort was great for 90 days after its first instantiation, then over six months its data became “suspect,” then after a year or two its data became prehistoric and close to useless.

The simple truth is, engineers (of nearly every variety) are toxically allergic to documentation. Asking engineers to have the human discipline of keeping up with documentation is like asking the sun not to shine. No matter what set of inducements is offered, or punishments threatened, the engineer simply remains toxically allergic to documentation and its maintenance (probably a DNA thing).

Industry tried to solve this problem by producing “discovery tools,” or tools that could determine automatically what our state of change was environment by environment. The idea being that if we ran discovery tools on regular intervals we could at least renew our CMDB baseline from time to time. It was an improvement, but discovery still lacks context for change and often is not thorough enough to generate real value beyond asset location and basic configuration.

Enter DevOps, which inherently has an artifact database capability to track versions of builds, deployments and releases over time. DevOps understands the need for artifact identification, articulation, compilation and history. Most early DevOps tooling has some form of it built into the products. But this is where a perfect storm becomes possible.

If our DevOps systems have achieved transparency; that is, if we are able to connect the context of change outlined from ideation (in the business requirements) through production release (including the history of testing, etc.) by version, we now have a great deal of wealth we could plug into a resurrected CMDB. And we could maintain the CMDB automatically (requiring no engineers to have to do manual things to keep it up to date). Our CMDB becomes a repository, or master repository of the artifacts of change we implement through DevOps.

This is more than just baseline or the ability to establish a “golden copy” of what change should look like. It is the ability to establish “relative” golden copies by development version while it progresses through environments of even highly complex SDLC systems. We have thus increased the value of a CMDB exponentially beyond its use in just production, but can integrate it, into the SDLC process and watch change become documented in real time no matter how fast or how often we move change.

This should be groundbreaking. Even if the CIO has burned all his funding and political capital on a former CMDB project that has had less-than-desirable results, he or she can salvage what is left of it and quietly use DevOps to resurrect the CMDB, making it an integrated step of moving change. Once the CIO has demonstrated success doing this, he or she can offer this reclamation project to the business and begin exploring ways of creating CMDB valuation to the business (beyond asset location and valuation).

What emerges is a DevOps-centric CMDB. DevOps, having become the singular path to production, has both insight and context into all change progression and simply updates the CMDB (master artifact repository) as needed automatically. In this way, an integrated CMDB becomes part of the DevOps continuum. And perhaps the greatest benefit or win is the simultaneous recognition of engineer’s allergy to documentation already baked into this sustainable solution. I would call that win-win—or, at least, a recognition of reality.

To continue the conversation, feel free to contact me.

March 28, 2016
Funding Failure by Intent

Why is it that when we purchase a lotto ticket, every part of our brain tells us we won’t win, yet some portion of it is able to inspire hope where there shouldn’t be any? Could it be the same part of our brain that inspires hope in the face of nearly certain failure has been nurtured by the software development projects we have been exposed to over the years?

It would appear that the ability to prognosticate has been the foundation of nearly every waterfall project that had a timeline of three years or longer. It is this myth of “certainty” that agile methods are designed to eliminate. In our new approach, we simply break down and digest the scope of work until it is small enough for our risk tolerance, and easy enough to implement in our abbreviated time horizons. We try, we evaluate and we try again.

But no matter how low we believe the risk to be (and generally the reward we expect from it), risk does remain. And now instead of a three-year project plan well-equipped with milestones, corrective actions and projected costs, agile gives us only the current sprint, based loosely on our historical velocity and a literal “addiction” to continue this process going forward without any real regard for a predictable end-point or total project cost. Continuous delivery infers continuous innovation (emphasis on the word “continuous”). And if done right, why would a business ever choose to end innovation that works?

Imagine the analogy of Mozart sitting down to compose a quick advertising ditty that your company could play as background music on its website. Mozart begins with a few bars, and notices a theme that is worthy of far more exploration … so he does. He continues exploring this until an epic symphony emerges. This was not the plan and does not meet the original requirements. This could be considered scope-creep in the waterfall world, and thus a failure, as no one stares blankly at a website for the 30 minutes it takes to hear an entire Mozart epic … or do they?

Every audience grasps the concept of background music we pay little attention to but subconsciously understand it improves our experience. But nearly every audience also realizes the beauty of a Mozart epic, and would gladly buy CDs, MP3s or concert tickets to hear one of those over and over again, relishing every note. The question this analogy begs is simply this: Would Mozart’s employer—more specifically, his development manager—have been free to fund this “failure” or not? What happens when a project changes direction significantly away from the first vision we had of it? Does DevOps enable us to pursue this radical change in direction, or are our legacy financial systems and methods binding us to a waterfall delivery and reporting structure—and, in so doing, limiting what we will ever be able to accomplish through radical innovation?

The beauty of agile construction and DevOps delivery is that we may deliver “unexpected” innovation. Nearly every time a user “sees” what we have delivered to meet their need, they immediately think of “x” more things we could do to make it even better. This may result in taking a project in a radically different direction within the confines of a single financial year. To further enable our ability to do this, our legacy financial funding and reporting systems must evolve to match this demand. Our new risk profile must include the ability to fund failure on purpose, understanding that failure itself is part of the maturation process of innovation, not the death-knell of the company.

A more honest approach to funding agile/DevOps efforts would be to determine the annual investment the organization wants to make in certain capabilities, and innovations it expects to see in each thematic area. Overarching themes are funded on an annualized basis; however, great autonomy is granted and expected within the themes to ensure delivery is achieved. Pushing financial responsibility down in the organization to as low a level as possible has the effect of pairing the team leaders with the costs of what they do. Spending too much time on delivery of any given feature or function may cause the team to re-evaluate the approach and propose a better solution. This is more unlikely when budget responsibility is managed several levels above the team doing the work—in that instance, the budget resembles more of an infinite resource they need not be concerned with, until it is too late.

When proponents of DevOps claim it is changing the culture of a company, they are not limiting this view only to IT. Tertiary services in a company, such as the supporting financial budgeting systems, hiring systems and even contract management systems, may require significant change to match the pace of delivery and innovation DevOps delivers. As an enterprise looks to implement DevOps, it should look broadly at what else may be required to fully enable the pace of innovation it is capable of.

To continue the conversation, feel free to contact me.

March 23, 2016
Interoperability’s Impact on Transparency

We fix what we see is broken. Providing “transparency” into the DevOps-enabled software development life cycle (SDLC) has tangential goals of quickly illuminating what is broken, running slower than it should or working flawlessly.

Automated testing systems, for instance, are generally good at highlighting pass/fail results from integrated testing in the development process. Once they have proven their reliability, they become a de facto part of the development automation and rarely are thought of again. However, when a post-DevOps legacy build begins to take longer and longer to complete, are we able to quickly examine the runtime of our entire body of integrated tests to ensure they continue to perform within expected execution timing parameters (i.e., could the code be fine and the tests themselves be the culprit)? And is that problem exacerbated when the number of different testing tools begins to explode on the same coding unit of work?

Connecting the dots has always been the goal of DevOps. In the ideal world, the business requirements—or the epics and stories—are part of the “dots” that remain connected or associated with each change in coding, builds, deployments and ultimately release. But are these “dots” easily transparent in each part of the DevOps continuum?

Transparency from multiple perspectives

To think engineers are the only audience of DevOps is too restrictive. Product and project managers are also interested in knowing “the state of change” as it relates to the business issues they are tracking, and in real time. Whether the requirement originated from a bug-fix or as a new feature, the audience of people who want to see what is where within the DevOps set of tools is not confined only to people within the development or operations of IT. The ability to provide sensible, near real-time presentation of what changes are where gives the entire organization transparency to the progress of change. It impacts every organization from customer service, to program management—executives want to know as much as casual viewers in many different business functions.

Audit is another key driver for full transparency. Regulatory compliance is at one end of this need; corporate quality standards should be on the other. Organizations always should be more concerned with quality standards than the regulations they are subjected to. Therefore, the process of audit always drives the need for greater transparency. Imagine the ability to have “push button audit” enabled by DevOps for any change under inspection. Imagine what further cost delta this represents to organizations that are still bound to only manual methods of examination. With DevOps, not only is this possible, it also should be normal. The entire history of any change entering production should be stored by each of the DevOps tools within the journey, from ideation to implementation. All test records should be immediately accessible (and automatically associated with the particular version of change in question). Everything from failed builds to the exact time it was released into each impacted data center around the world can—and should—be available for a consolidated, push-button recollection for audit. However, the ability to make this happen often is directly proportionate to the interoperability of the DevOps tools used.

Transparency of key data and processes

When point solutions are selected for the DevOps tooling (defined as best-of-breed tools, with no inherent interoperability with other tools in the process), the particular functions they perform are thorough and generally complete. However, the only way to integrate key information between different tools is through APIs and/or a homemade amalgamation of the data from each point solution tool into a common dashboard. Amalgamating data is always a challenge, particularly when you consider the point of view of the audience varies by function and business discipline. If you eliminate or reduce the points of view down to only engineers (as is what usually happens), you destroy the value of DevOps transparency for every other business discipline in your organization. If you spend the labor to create a very functional, multi-perspective dashboard, your investment can be much higher than what you make in the tools themselves.

Integrated solutions—generally pushed by the same vendor as the preferred (albeit more license-cost-heavy) solution—are not the ultimate panacea, either. While a given vendor may be able to offer tools that track requirements, construct builds, integrate testing, execute deploys and orchestrate releases, more often than not, the vendor is stronger in one discipline over the others. They may ask you (the customer) to use a limited number of tools designed for one thing to do more functions than the tools were intended to perform—which creates a recipe for constant customization and lucrative service deals for the vendor.

What becomes important to measure in any of these instances, is the level of inherent interoperability that will exist between each type of DevOps tool in the offering. If the “connecting of dots” is inherent, and the state of change can be viewed in any of the tools, that is a win. If the point-of-view offered extends to more than just an engineer, but includes other business audiences who may have interest, that is a win. If the reporting is push-button, that is a win.

Transparency of tool usage and roles

But beware the dark side. Whether your organization relies on point solutions or integrated solutions, both can be corrupted by an inconsistent methodology of user profiling within the tools themselves. For example, let’s assume we have a business requirements tool, a build tool, a deployment tool and a release tool. Generally, a programmer, who we’ll call “Dave,” has a very specific role in the SDLC process that doesn’t include releasing code (programmers usually don’t release code into production). Generally, the level of granularity in defining tasks and roles within a given tool is proportionate to the rules your auditors require (whether in-house or regulatory). So for a given bank, we might define the role of a programmer “Dave” to be able to see business requirements, execute builds all day long, deploy them to the test class (or first level) of environment in the SDLC and then simply watch what happens as it progresses to production, never directly interacting again.

One of the biggest problems DevOps tooling has is that each tool handles user management separately. That may allow me to define the roles of a programmer properly within each tool, but it does not enforce that my guy Dave is properly mapped to that role. I could make Dave a business analyst in the requirements tool, a programmer in the build tool, a deployment manager in the deployment tool, and a release manager in the release orchestration tool. This would allow Dave god-like abilities in each tool within the process. He could, in fact, create the requirement to move all our money to his account, build the code to do it, deploy it and release it into production. Dave would not have triggered any alarms in the tools, because each tool thought he was doing what he was supposed to be doing in that role. And I have never seen tools audit what role Dave is supposed to be mapped to given his job in the company or even spot-checking what role he carries in other DevOps tools to see if there is an inconsistency. Perhaps abstracting user management from the entire set of tools may be the answer—but then, who offers that capability today?

Transparency, when properly managing access controls and particularly at the push of a button, can provide solutions—even if only after the fact—to a great number of problems that exist in our organizations and in our SDLC process. We fix what we can see is broken. If our DevOps solutions can highlight transparency from ideation to production release, we can have confidence that what we see is real and worth our attention. When our transparency varies by tool, or loses connection within the process, our confidence of the state of change diminishes and our certainty about what needs our attention goes away. Using DevOps to shine a light on our true priorities is yet another cultural change enabler we have the responsibility to pursue.

To continue the conversation, feel free to contact me.

March 17, 2016
Changing the Culture of Metrics Analysis

You could say the development and operations teams gave birth to the idea of DevOps, as it is believed the chief benefits are garnered in those two functional disciplines. Because of this, metrics nearly always are conducted from the points of view of cost reduction, speed and efficiency. This makes perfect sense when business treats IT (and IT sees itself) as a service- or cost-based organization. But what if DevOps has a few more characteristics that could actually change that culture? What if, for the first time, DevOps could enable IT to become a real partner with the business organization it serves?

To make this happen, business and IT must begin to share attainable goals. Imagine the impact of a highly functioning, enterprise-class DevOps service within IT; upon the time to market of products and services the business is looking to launch. A proven, consistent ability of IT to deliver on time or ahead of schedule (i.e., reliability) then begins to change the dynamics of what marketing proposes and delivers. Time then becomes a valid metric to consider when looking at customer adoption rates. DevOps usually does not directly impact initial market demand, but the impact of delivering new services or features months or years ahead of previous projections can dramatically impact time to market. For example, if your business is able to change its billing model or billing methods six months to a year ahead of every one of your competitors, what impact might that have on market adoption? At that point DevOps becomes a key business differentiator, until the rest of the universe has it.

Consider an additional impact of a highly functioning DevOps service on feature delivery or the feature road maps themselves. The implementation of DevOps inherently delivers smaller units of change on a higher frequency, reducing risk and allowing iteration and alteration of future plans based on the real-time feedback of change success. Marketing focus groups may provide an original road map, but the impact of DevOps can be integrated back into those same focus groups. Instead of imagining what might be possible, DevOps allows delivery of what is possible to see, feel, touch and prompt further imagination. The net impact is an increase to the pace and depth of innovation itself. You know what your customers really want, and you can give it to them quicker. What kind of impact does that have on the reputation of your business if it becomes a market leader because of this? Or what happens when a large organization is then able to deliver with the speed, agility and flexibility of a small firm?

Cost Reduction vs. Revenue Generation

Now consider the value of cost-reduction metrics compared with revenue-generation metrics. It is easy enough to determine that the pre-DevOps delivery cycle might have been 12 weeks per feature as an example, or that each build took six hours to complete and now post-DevOps takes only 12 minutes to run. Metrics like these will begin to show an increase in productivity by existing development staff—or, maintaining the same feature delivery rate, a reduction in the number of personnel needed to achieve them. Similarly, it is easy to show a reduced number of trouble tickets post-DevOps, and a concurrent reduction in the average time to resolve each ticket. DevOps then has enabled the CIO to continue to take arbitrary budget cuts demanded by the business each year. This is not partnership—over time this is a recipe for the reduction, pace and quality of innovation (the smartest people leave first during layoffs; they don’t need to stay).

However, even significant cost-reduction metrics pale in comparison to revenue-generation metrics. As an alternative, imagine what the net growth in customers of a given product might be, over a six-month period of time, because a desired feature set was delivered early? And what if it is two to three years before your competitors can implement the same high-functioning DevOps services you already have in place today? If a business makes $2 per-widget profit and captures 20 percent additional market share over six months, equating to “x” million new customers … well, you begin to get the idea. All of the sudden IT is no longer viewed as a cost center by business; it is viewed as a revenue-enabler. The immediate business response is, “If I allocated more budget into the types of staff facilitating DevOps services (running the gamut from coding to operations to marketing), could I gain even faster delivery rates or deeper feature sets or more market share by pioneering what has never been pioneered?”

The two main reasons this realization hardly ever occurs in business today are that 1.) IT has failed at delivering on promises so often it has trained the business to believe it “cannot” deliver on time or on budget; and 2.) few businesses conduct accurate time-to-market metrics measuring the delta of what additional “time” does to the equation, largely because they never had to. DevOps provides the catalyst to begin to re-examine both of these phenomenon.

But it goes deeper than that. Changing the culture of innovation, in which I am eager to test new ideas faster, means I need more ideas all the time. The competitive advantage our nation, our businesses and our careers have always benefited from is our ability to create and innovate. Other nations around the world, or other organizations your company competes with, may be able to do repetitive tasks faster or more efficiently. But rarely do worker-drones create or produce a wealth of ideas. But if we begin to focus on the long-term impact of fostering innovation in our own country, and in our companies, we will remain perpetually out front. The goal of DevOps is not merely efficiency, it is also to spark the imagination. When our focus and our metrics reflect this, we all win.

If you would like to continue a conversation on how to create and implement a highly functioning enterprise class DevOps service, feel free to contact me.

March 14, 2016
Measuring the Cost of Service Creation & Adoption

Adopting and implementing DevOps isn’t an overnight project. It takes time and effort to determine how it best fits into your organization—and what benefits your organization will realize as a result.

Creating an enterprise-class DevOps service is typically done in one of three ways, depending on the organization, its leadership and its employees. Each has its pros and cons, and each has a number of factors to consider.

A Fully Centralized Model

In this model, tooling decisions are done in a central service provider group for the entire organization. Tools are managed centrally and standards are created centrally. DevOps automation creation (at least the templates or frameworks) is done centrally. This model is highly advantageous for organizations with a wide variety of technology platforms (think large population of legacy apps) and use many technology components for their apps, such as multiple types of databases, middleware and operating platforms, for example.

Not many large organizations are ready to scrap what they have in the application portfolio and start over using the cloud and containers and develop in a singular methodology. Using a centralized model, companies can continue to improve upon their years of investment in legacy applications and adopt a DevOps continuum without starting over from scratch.

This also is the lowest-cost model, as tooling is standardized and administered from a central point and licensing can be negotiated to maximum effect. In addition, labor (generally the highest ongoing cost of any service creation and implementation) is minimized by centralizing it. Training costs also can be minimized by establishing a centralized training program for all impacted audiences. Companies can change their incentive programs to achieve hiring objectives and business results as priorities change—something that isn’t practical in a decentralized or federated model.

A Fully Decentralized Model

In a fully decentralized model, tooling decisions are determined and managed by each development team. This has the advantage for shops with small numbers of applications or singular methods of development—fewer applications are better candidates for starting over with cloud or container integrations and can take advantage of the newest technologies from the starting point, if needed.

Shops such as Etsy or Facebook seem massive, but from a user point of view, they really only present a single application that needs to be maintained and improved over time. Instances like this may favor a decentralized model because the actual number of application candidates may be highly limited—sometimes only one.

This model typically represents the highest costs, as repeatability and standardization can only be measured at the development team level. As such, this is practical for a very limited number of applications, or technologies; if those numbers grow, the strategy quickly becomes untenable. In addition, this strategy has an uncanny history of pulling development resources to do the DevOps automation engineering that needs to be created and maintained. Typically, the highest skilled developers are assigned these types of duties, and thinking in terms of templates and frameworks is rarely their first approach. Developers tend to be expedient, and believe they will always be there to make sure it works. This also can reduce the potential for repeatability and predictability (raising costs).

A Federated Model

In a federated model, tooling decisions and standards are done centrally, but management and operations of the tools are performed by a limited number of subservient organizations (often by line of business). Centralized templates may or may not be viable depending on the variability of the business units they serve. The chief advantage of this approach is the segregation of tooling and reporting by business unit (typically) without having to build that kind of segregation in to tools that were not designed that capability in mind.

This model also has the advantage of increasing the speed of adoption, as several parallel efforts can take place at the same time and spreading engineer resource demand across multiple organizations. It typically makes the most sense in larger organizations with multiple business units whose application portfolios that would benefit from an onboarding to DevOps.

What’s more, a federated model may be the most cost-efficient. Labor costs will be inherently higher as functions are replicated from business unit to business unit; however, that replication will result in increased adoption rates and project completions (you get what you pay for). If business units are able to leverage centralized templates for each technology type, adoption rates become an order of magnitude faster (or have that potential). Tooling costs still can be negotiated centrally, even though management will occur in the business units. Again, there is hardware replication from business unit to business unit, but the ability to segregate users, permissions and operations down to the business-unit level can save tremendous labor costs if development was the only other option to create that capability in tools not designed for it.

Adoption Costs

Measuring the cost of adoption requires focus, no matter which method you employ to create the DevOps service. Applications moving to DevOps are said to be onboarded (where the original build, deploy, test and release automation is created). To successfully onboard an application requires education of the sponsoring executives, training of the impacted developers and testers, and coordination with the change and release managers.

Collection of meta data about the behavior of application components is no small task, requiring frequent follow-ups and action items to reach completion. The effort to onboard applications to DevOps becomes the first cost for estimation and tracking as the application portfolio is attacked.

Even after onboarding of a given application is complete, costs continue. The terms “continuous delivery” and “continuous integration” should also infer that application development methods “continuously” evolve and may adopt new technology components. Significant changes may occur that require revisiting the DevOps automation to adapt to these ongoing revisions and updates.

While this work is rarely as extensive as the initial onboarding, it is measurable. Work of this nature is usually referred to as BAU (business as usual) and represents the second significant cost of DevOps services that must be measured and tracked. If no thought is given to this work effort, inevitably the team assigned to do onboarding will be distracted by the BAU requirements until they reach the point where no new applications can be completed.

Staffing cost estimations can be presented at the component level for both onboarding and BAU efforts. For example, if one DevOps engineer can onboard 20 components per week (assuming familiar technologies within a given portfolio), and if the general ratio of components per application is 4:1, then a given DevOps engineer can successfully onboard five applications per week. If the engineer is never distracted with BAU, this can continue until the entire portfolio is onboarded. However, my experience has shown that for every 20 applications completed with onboarding, the BAU work effort can absorb nearly a full DevOps engineer. So if you start with a team of 10 people, you will onboard only 200 applications total before the entire team is required to focus on BAU work effort and has no more bandwidth to continue onboarding new apps. Understanding this demand, and learning how efficient your particular engineering team is, will be key to managing costs while completing the onboarding efforts of large application portfolios.

March 8, 2016
DevOps and the Near-Decapitation of Variability

Not all automation scales. One of the chief impediments of delivering an enterprise-class DevOps service is the decentralization of tooling choices—or worse, the methodologies within the tool for delivering CI/CD. If the methods used to deliver automation are as varied as the last engineer who created them, the spaghetti that ensues will not scale much farther than previous methods did.

The trick to scalability is not so much tooling-dependent as it is approach-dependent. Templates or frameworks (or any word you would use) to describe a relatively preformatted approach to automating a particular kind of technology component are the key to achieving scale.

Ideally, your highest-skilled engineers create the basic template or framework that other, lesser-skilled engineers will use to create the actual automation for each system as they move forward. Using a single way to automate a particular kind of technology enables you to manage large numbers of systems utilizing that technology from a single point of administration. For example, a standardized template used to create the automation to deploy a MySQL-type database could be altered over time to include new testing requirements detected from results found later in the QA process.

Once the new testing is incorporated into the standardized template, the resulting automation is fixed once for every instance of that kind of database in every system that uses it. Otherwise, you’ll need to remember to update every script for every instance of a system that uses this kind of database. Those updates likely will be implemented differently across the applications by different engineers, which may negatively impact testing results (thus defeating the purpose of testing consistency, and actually generating more unnecessary work to locate the variations).

Templates or frameworks (or pick the synonym that suits you) can reduce variability in “how” the automation continuously delivers and integrates. Once variability is killed (or nearly killed), predictability can emerge. Predictability throughout the entire SDLC lowers overall costs, as finding and fixing bugs becomes more systemic. What’s more, there are fewer bugs, thanks to battle-tested processes that produce consistent behavior. If the tooling is sophisticated enough to offer visibility, the costs of quality are reduced even further, and the speed to deliver is radically increased.

DevOps promises that combination of reduced failure rates with increased speed of delivery. However, embracing unstructured variability in creating the automation can kill it. The Ops side of the DevOps equation is all about doing things systemically and predictably, as that is the only way to maintain stability while changes whiz through their world. Getting Ops to buy into DevOps is much easier when they have complete visibility into a predictable, repeatable, process for delivering change. Avoiding that 3 a.m. phone call is something everyone has in common, and reducing variability by focusing automation through the lens of templating or frameworks is an excellent way to achieve it.

March 4, 2016