devopsy.kinsta.cloud

Author: Kristian Nelson

  • DevOps and the Identity Conundrum

    DevOps and the Identity Conundrum

    “Who am I?” It is perhaps the most basic question of consciousness. The answer could be subdivided by aspects of my life story … my career, my family, my beliefs, my education … the list could continue. But since I am unique, the challenge in this digital age is for me to somehow remain unique, and whatever descriptions are used to catalog who I am, I must remain at the center of them all. If my identity is stolen, how does anyone know, that I am who I say I am?

    This question is fundamental to every interaction I have with the digital world. And unfortunately, every system I interact with looks for me to redefine who I am, over and over again for each of them. I create a Facebook profile, a Twitter profile, a LinkedIn profile … and the list goes on again. But yet another weakness of our digital age is that there is no cross-reference between these systems to ensure I am who I say I am. This phenomenon remains true within DevOps tooling as well.

    The Career Identity Conundrum

    Let’s focus on only one aspect of who I am, my career. Who my current employer is dictates what electronic business systems and networks I should have access to. Switching jobs to another employer changes—or should change—the access I have. But getting more specific, what my role is at my current employer should also have an impact on what tools I have access to, and what I can do within them. And as my role changes, my abilities within a tool should change, as well. If I move from practitioner, to manager, to director, my abilities within a given tool should probably diminish, holding practitioners accountable for actions rather than superseding their responsibility by doing things myself.

    Then there is the area of focus I work on. Business segmentation differentiates my work from product or service “A” in contrast with product or service “B.” My role might be an application programmer, or database administrator, but then “who” I focus on within a company is yet another level of differentiation. The business may prefer that teams of people focused on product “A” are restricted from making substantive changes on product “B.” So two different database administrators need two different levels of access within the same tooling, segregated by line of business or product line to accomplish it.

    The Career Consistency Conundrum

    The problem with my corporate identity in the digital age is that it is fluid, like life. In the course of my career I will change employers at one time or another. But beyond this, I will change roles, and/or areas of focus within the same employer over time. I may be assigned to work a “special project” in addition to my normal duties during my time at a given employer. And where DevOps relies upon several types of tooling to facilitate the continuum, this fluidity in my career makes it difficult to keep up with, and can drive administrative costs quite high.

    For example, if I am an application programmer (normally), I will then have certain abilities within the build, deploy and release tools—likely high ability in build, moderate in deploy and low in release. If I am temporarily assigned as a release manager for a different product/service (in addition to my normal duties), it should radically alter my abilities within the DevOps tools, at least for a short time. But when I return to normal duties, the only mechanism that gets my abilities reset to what they should be is a strong process control system. The tools do nothing to facilitate this (no reminders, no examination of HR systems for roles, no look at each other to see what I am supposed to be).

    An Identity Architecture

    It occurs to me that the constants in resolving these kinds of problems are time and who I am. If I could hold an identification construct that identifies myself in career terms—which employer, which role, etc. and for how long—that identity construct could be employed into every system that accepted it. In effect, it would be the ultimate abstraction of user management, transferring the burden from companies and HR departments to the individuals who know, and need to know, the end user.  While companies would still need to interact with my personal identity construct for verification, assent and record keeping, having all electronic systems accept my ID construct takes all of the work out of programming those interfaces, makes permissions and roles common, and keeps up with life’s fluidity.

    In addition, who I am becomes the same in every system that accepts this construct. I cannot define myself as a release manager in one tool and a test manager in another and an application programmer in yet another to give myself god-like abilities in the DevOps tooling at my company. Each tool would examine the same identity construct I hold and receive from it, my normal duties, roles, permissions, etc. I would have a consistent profile across corporate tooling that would reflect my latest roles and use dates (beginning and end) to validate me against their own records.

    An Identity Lockbox

    The trick is, how do I keep this identity construct from being stolen? I believe this is where biometrics begins to play a key role.  Fingerprint technology is commonplace now, even in cell phones, which are usually chained to us like an additional finger or toe. We rarely go anywhere without them. And when carrying them, we use features such as Touch ID to replace the manual keying-in of passwords. If my identity construct was paired between information likely I alone would have and biometrics tuned only to me (retinal scans, fingerprints, etc.), the pairing could unlock the construct for either use or revision.

    Stealing the construct does little good, as without the pairing biometrics, it could not be used or revised. The applications for an identity construct like this are limitless—everything from securing the right to vote, work and collect social security to validating credit card purchases and opening up new social media accounts.  If Facebook employed (or accepted) this kind of construct, it would be more difficult to create spoof accounts of me, as the unbeatable pairing could decipher who really owns a photo, or document, etc. And looking forward, a standardized identity construct such as this could interact with the emerging world of IoT (internet of Things), where devices could know a lot about me, and know it was truly me.

    Impacts to the Bottom Line

    I am not suggesting some wild end-of-the-world scenario where computers control our every move. We already have that in some form or another (see banks – ha ha).  But I am suggesting that in the interest of reclaiming digital privacy we should be moving the ownership of identity constructs back to the people who own them: us. We should each own our own singular construct, keep it up to date and as filled out or complete as we want to, sharing only what we want to. The chief benefit of this would be an increased difficulty in stealing them from us. We can all skip the 666 tattoos, and instead try to reclaim a moniker of privacy by holding our own digital identities ourselves. Using them like a master key that will open systems we “should” have access to and abilities we “need” to have to perform the work assigned to us.

    The upside to corporations is increased security that requires less on their part to maintain. A single set of records with start and end dates validates IDs used or attempting access. The upside to the software industry as a whole is a common interface for logging into an application with discreet identification. It removes the doubt about “borrowing” an ID if unlocking access requires a biometric pair as well. It would allow every application to abstract user management and make it common. Roles would be abstracted. Permissions could remain unique to each application based on their interpretation of the role.

    Moving to a self-maintained identity construct could be taken further for benefits in other areas of our lives. While we have discussed some of how the career portions of our identity might be subdivided and cataloged, imagine for a moment the benefits of having our heath characteristics attached to the same kind of identity construct. Family records, memberships and associations … pick your classification of how a person is described for who they are. Having a way to retain ownership of those things in the digital age is key to keeping our unique position in the world unique. Losing control of those things becomes catastrophic. Where DevOps benefits from a construct such as this is but the tip of an iceberg compared to the benefits society reaps—or at least, our digital society could reap if we pursued it.

    To continue the conversation, feel free to contact me.

     

    — Kristian Nelson

  • DevOps and Automation Abstraction?

    DevOps and Automation Abstraction?

    Similarity breeds consistency, giving way to predictability and, thereby, dropping costs of operations and quality from a lower number of incidents and a shorter duration to resolve them. But at the heart of this chain of events is the recognition that we tend to do certain tasks the same way, over and over again, for a given type of technology. Variability then becomes our enemy.  Consistency becomes our ally.  So the question: How do we as DevOps service providers instill a level of consistency in what we do that is not engineer-specific (i.e. only one guy knows how it is supposed to work) and will still scale at the enterprise level?

    From a people perspective, we use teams, whether co-located in person or virtually using the same backbone of tooling over wide geographical separation. From a process perspective, we use governance and education to ensure the same processes and approaches are taken at a fundamental level across different types of technologies. And finally, from a tooling perspective, we use templates or frameworks (or pick your synonym) that allows us to construct a shell that guides the automation construction to be consistent from parent to child. The combination of these three fundamental elements leads to consistency that scales. But is that all?

    What Are We Missing?

    Could there be another fundamental element we have overlooked—namely, abstraction? Case in point: Consider the effect of a drag-and-drop interface versus a typed one to accomplish a given task. Both look to accomplish the same outcome. One could argue the typed text approach offers more flexibility and customization to get the task done. But the drag-and-drop method offers a consistent, simplistic way of doing a similar task, the same way, over and over again. Examining both methods objectively, the drag-and-drop method offers a level of consistency the typed method never could. For the typed one, you would always have to check to ensure punctuation and lack of changes (additions or deletions) to “know” it was the same script as last time. The drag-and-drop method on the other hand, is “canned” or largely unalterable; it may limit customization (by design), but it excels at consistency.

    The drag-and-drop method achieves this consistency by effectively abstracting the typed or scripted code within it from the user who employs it. The user only makes the choice to use it, when desired (or needed); the rest of the mechanics are abstracted from the user. The reason drag-and-drop methods exist is beyond just consistency, however; it is for simplicity. Apple computers would appear to be founded on this idea. For those of you old enough to remember DOS on PC computers, the parallel here immediately becomes clear. DOS was a typed-based command-line operating system (OS) fully capable of operating a PC many years ago. But Apple’s graphical interface made the same functions “intuitive” and forced the PC industry into Windows to stay competitive. Nobody chooses a command line OS like DOS when something as elegant as iOS exists, or Windows (or pick your graphical poison here).

    A graphical OS abstracts core functions from the user, and makes every attempt to make an elegant intuitive packaging of the same functions. Users “get” an iPhone, which accounts for its sales. You don’t need formal education classes or an eidetic memory to use an iPhone. Users effectively have been abstracted from all the typed code it takes to create an OS like this, millions of lines of such code packaged up into swipes, beeps and elegant graphical displays. Keep in mind, at its core this OS abstracts users from variability that might exist if hundreds of companies created their own versions of iOS. The simplicity is paired with consistency, and the quality of Apple is a result (for better or worse). But whatever the quality of Apple is, it is far better than if a wide degree of variability were permitted in the vendors of iOS or the construction of it.

    Extending Abstraction in DevOps

    Now enter DevOps. Containers are a level of abstraction widely popular in our discipline for keeping hardware particulars away from what are typically software engineers. The value of this abstraction is well-understood. My challenge is simply to extend this thinking into the next tier of DevOps tooling. What if the ability to include database DevOps automation was simply a product of drag-and-drop methods that every build tool understood perfectly? Granted, you may have to limit the expression of this idea to say Oracle or SQL Server at first to capture the nuances and particular capabilities of products and technologies in this space. But if someone built a “container” for Oracle, I would wager the choice to use Oracle would rise substantially over its competitors, at least until those containers or abstraction layers were just as effective with other database products.

    The abstraction phenomenon need not be limited to hardware, or as I have proposed to database platforms. It also could apply to .NET or Java development efforts at the top of the stack. The steps needed to take an idea, code it and put it through the build process are going to be largely similar when working in a .NET shop or a java shop (or pick your particular development tool of choice here). Even the top of the stack—that is to say, the development of applications itself—could be a viable candidate for abstraction from the mundane details as it relates to a build construction or a deployment to a given class of environment.

    Creating effective containers for the application layer in effect limits variability, and increases predictability. I realize it has to be technology-type sensitive. The .NET shops have their own container. The java shops have theirs, and so on. The only question is whether we are able to include enough of the common core steps we take to produce a build with a given technology type to form an abstraction construct of value. Over time, this would include every type of technology widely in use today.

    The effort to simplify through abstractions ultimately would result in our ability to include technologies such as WebSphere at our pleasure. It would also allow us to switch out WebSphere for a new competitor using the same abstraction constructs, universally, just by rebuilding all our apps one time. This kind of power is alluring to development teams. It drives application developers and operations engineers to begin to work together to look for the most cost effective technology that can perform a given spec. Being the best at something is no longer a cage from which there is no escape later.

    Technology components would be selected based on performance and cost of ownership (i.e. operations) later. Switching is ultrasimplistic if all we ever used to include it in our build routine was a drag-and-drop style method with universal, well-defined parameters. A customer’s ability to switch products to the best of breed through easy or simplistic means keeps pressure on vendors to actually be the best of breed. If the performance suffers or a new release is full of bugs, developers just switch it out through abstraction (not long, complicated scripts where everything has to go right, including a full moon on the third Tuesday of the month with a Gregorian chant in the background).

    Impacts to the Bottom Line

    The selection of DevOps tooling often is a cantankerous one in large organizations. Debates ensue about flexibility, extensibility and whether a given tool can be customized to accommodate the unique little snowflake of my development teams. Perhaps these debates are headed in the wrong direction. Perhaps instead of prizing and valuing the extensibility or customization a given tool affords, we should be looking closer at how that extensibility is achieved.

    For an enterprise to make an unwitting choice to get into the script business, is effectively stating: “I will be maintaining the scripts I use to construct a build with as much rigor as I maintain the code these scripts are designed to create builds out of.” So, we just doubled our programming staff without realizing it—one half working on the application code and the other half working on the scripting code that “automates” the delivery of a build during construction. While overall speed may be increased, it comes at the cost of doubling already expensive headcount.

    As an alternative, what if enterprise customers were asking tooling vendors how simplistic their tools are to use, and how standardized are their approaches. While we may not enjoy losing a degree of flexibility as we may be unable to customize a drag-and-drop method construct, the payoffs to the business are simplicity of use and scale at an enterprise level. Not only do I refrain from doubling headcount, I can actually free resources to focus on the disciplines and areas they enjoy most instead of performing script maintenance. Abstraction as a construct: From the hardware layer up through the application layer, it may be a driving fundamental that keeps DevOps scaling and lowering the cost of innovation for many years to come—but only if you begin to ask for it.

    To continue the conversation, feel free to contact me.

  • Putting Ops Back in DevOps

    Putting Ops Back in DevOps

    What Agile means to your typical operations staff member is, “More junk coming faster that I will get blamed for when it breaks.” There always is tension between development and operations when something goes south. Developers are sure the code worked on their machine; therefore, if it does not work in some other environment, operations must have changed something that made it break. Operations sees the same code perform differently on the same machine with the same config, which means if something broke, the most recent change must have caused it … i.e. the code did it. The finger-pointing squabbles are epic (no pun intended). So how do we get Ops folks interested in DevOps without promising them only a quantum order of magnitude more problems—and delivered faster?

    The problem is exacerbated when the topic of DevOps comes up, as it is most often a developer doing the talking. And the topics under conversation are things every developer would have substantial interest in and pretty much nothing that would interest the operations part of the equation. Many Ops folks feel as though they have been dragged into this DevOps phenomenon by osmosis, without their consent, and now they are supposed to be reaping huge benefits from it (by magic, as near as anyone can tell). Developers working faster is traditionally not something Ops folks look forward to; rather, it may be their worst nightmare. So, if the advocates of DevOps all tend to be developers, how do we get operations folks to become our evangelists as well?

    QA Impacts on Evangelism

    It begins with quality—not the ethereal concept of quality, but rather the specific responsibility for who owns it. Finger-pointing be gone; developers now own QA. If the software development lifecycle (SDLC) environment is broken, developers own it. Ops can sit back, toast marshmallows and enjoy the show. Developers should be including whatever configuration file changes their application needs as part of the deployable package. So nothing is done manually anymore. This means no one could forget something, as machines do not forget; they do what we tell them. If by chance, some random operations staffer decides to doink with a config variable, the deployment software should simply overwrite the manual change with what is in the deployable package created by the developer. Even malicious changes can be overwritten, so now, all change is included in the development class environment in the one and only deployable package. That is the only package that moves forward.

    If developers own the QA, there is now a shift in the force: The operations team has a new role, one of coach and adviser, instead of doer. The software now is the doer, the developers are the packager and the operations staff are the witnesses of how it works. If it breaks, we all know who must fix it (and who is to blame). But once blame has been permanently affixed, something else changes, namely the culture. Now that the operations team does not take the blame, it remains interested in seeing its operations run without error. This allows operations to take an active “helping” interest in getting things fixed faster. It does not really matter who is to blame, once that argument has been settled; now it only matters if they can make something work. Harmony between factions begins to emerge.

    Scalability Impacts on Evangelism

    It continues with scale. Self-service is the nirvana of your average operations staffer. If the responsibility for provisioning can be moved into a push-button self-serve automation, nirvana will have been achieved. Imagine the offload of repetitious work tasks that consume the life of your average operations staffer. Instead of setting up (and taking down) environment after environment of one class or another for one application or another, requesters can all do it themselves with the push of a button. Even the accounting or financial impact is something the accounting team can guide the user on; operations does not have to worry about the cost of its operations anymore (or, at least, worry less). If too many environments are being provisioned, the accounting team has the power to change the rules and keep it from happening automatically without operations having to be the bad guy.

    At this point, Ops can resume a focus on something it tends to enjoy more: monitoring. Ops staffers enjoy being able to provide real-time reporting on where things stand at any point in time. It is even more rewarding to provide more meaningful information such as who is using this application, what users are doing with it and when are they performing that activity. To monitor that kind of information, operations must partner with developers at the outset of a project to determine what kind of application logs are needed later for monitoring. Without the daily barrage of mundane provisioning tasks, an operations engineer is free to focus on more rewarding (translate: higher skill-level) tasks. The days of changing tapes for the backup may have reached their end (or pick your similar set of mundane tasks which DevOps automation can fully eliminate).

    And Speaking of Monitoring

    It merges into the future. How do you create an operations staffer as a DevOps evangelist? Start by exploring potential that only DevOps can enable. All those nagging alerts that are created when an environment goes down while a deployment is made … all of that could be turned off by the DevOps automation, then turned back on when the deployment is completed. Again, it all would be machine-driven, with no human mistakes anymore, ever. This will begin to cut down “noise” in monitoring/alerting systems into only useful information. However, integrating monitoring systems into the normal DevOps continuum extends beyond just an on- and off-switch.

    Forward-thinking organizations begin to consider testing and monitoring as two parts of the same evaluation discipline. When a test is needed and when a monitoring characteristic is needed now can be determined in total rather than just in a development capacity or operations capacity. A holistic view of what can and what should be evaluated about an application, and how that will be achieved, brings the operations team to the table with the development and testing teams to plot holistic solutions that will live on beyond the development project timelines. Examining monitoring relative to version-controlled artifacts allows for organizational constructs within the monitoring discipline that never were possible before DevOps enabled them.

    The Masters of Hardware Abstraction

    Containers, or similar types of technologies, abstract an application from the hardware details it runs on top of. Technology such as this is a trend in DevOps that greatly simplifies the world of hardware from your average application developer. So who will retain the knowledge about the jungle that lives and breathes underneath? Ops has an extended role in understanding what lives underneath the abstraction layer that lives on top. Over time, only Ops will understand these particulars. They will become the only in-house experts about which cloud provider to use, which sets of physical hardware performs best and under what conditions.

    It is this kind on unique expertise that makes an operations engineer extremely valuable (translate: pricey). It is the kind of knowledge careers are made upon. It is ageism-neutral. Older employees do this today, and they will continue doing this in the future; the only difference will be fewer and fewer people will know how it is done. If the domain of application development is the province of our youth, then the domain of hardware abstraction is easily the province of our more experienced employees. These realities create personal value for our operations teams, and they are only possible in a post-DevOps embrace.

    Impacts to the Bottom Line

    The adoption of DevOps by the Ops portion of the equation is critical to overall success. Passive-aggressive resistance by operations staffers can delay or destroy altogether the value of DevOps. Overcoming this is a first step. But creating operations evangelists goes an order of magnitude further: With active operations staffers becoming internal DevOps evangelists, the entire operations team can begin to look forward at what DevOps will ultimately enable for them. The value they will carry back into development efforts becomes substantial and the entire company benefits. Careers can emerge that were not possible before it—and, just as importantly, so can profits.

    To continue the conversation, feel free to contact me.

  • DevOps: The Glue of Workflow

    DevOps: The Glue of Workflow

    The best tool in the DevOps continuum is the sticky wrapper that holds it all together. It is the workflow that binds it and makes sense out of the term “continuous.”  CIOs often are enamored with a build or deploy tool, but it is the workflow tool in every DevOps strategy encapsulating more than one function that is the real hero. Imagine having the perfect test automation scripts but no way to execute them outside of constructing a new build. Or imagine having the perfect CMDB integration scripts but no way to call them outside of the build process. Or even worse, imagine having the need for human approval before a deployment takes place (or is promoted), and no way to integrate it outside of a custom deployment process.

    Lean DevOps Construction

    Attempting to make very few tools perform these functions without a workflow engine ultimately forces automation engineers to create functionality within a tool it was never designed to perform. Conversely, when a workflow wrapper is available, build and deployment processes can remain small, elegant and consistent, allowing the workflow engine to integrate extended functionality into the continuous delivery cycle.  Size of the automation constructs you create does matter: the smaller the better.

    Using Lean methodologies in your DevOps services, it is better to break down automation constructs into the logical functions they are intended to provide. Build automation, for example, should not have to include extensions to turn monitoring functions on or off. Build automation also should not have to worry about the deployment process; its primary singular purpose is to construct a new build. Where that new build is then deployed becomes the single purpose of the deployment process.  Attempting to automate the entire cycle of an application into one automation construct from end to end (instead of smaller blocks that work together) increases both size and risk (as well as length of time to execute) of that one event.

    Dealing with Humans in the Automation

    Human integration is the bane of DevOps processes. Every time you have to pause automation to obtain a human approval, you introduce time delays that cannot be predicted or occur with consistency. Imagine a deployment process that requires a human to press “yes” before continuing. The introduction of a human approval can sit idle on a tooling screen for hours during lunch, or unexpected days off, or vacations (when others are afraid to take responsibility for passing a substandard build, even for further testing). Deployment tools can integrate human responses, but this is not their forte. They are generally ill-equipped to use multiple contact methods, go up the management chain when needed or ping a mobile device when preferred by the user listed as responsible. Workflow tools are ideal for all of these human-contact logistics.

    The challenge during DevOps service construction in utilizing a workflow wrapper is keeping human interaction to the absolute minimum required by outside regulating agencies and still achieve compliance. The alternative is to automate processes with such quality and consistency so that human interaction would merely become redundant and unnecessary. The workflow wrapper is not an excuse to simply automate what human interaction you have today. Instead, it is a highly functioning tool to allow you to redesign more-effective automation processes, limiting human interaction where possible. The goal is to build small, independent automation constructs that work together in a highly efficient manner.

    Hold the Presses

    Communication or notifications also can be handled in a workflow wrapper much better than in individual DevOps toolings. Imagine for a moment that I am a release manager, wanting to know the progress (or rather obstructions to progress) for my new application. To accomplish this notification process, it would have to be embedded within my build process (to know if it passed or failed). The alternative is that my workflow wrapper checks the build logs and can send me notification in the method I prefer (email, text message, an Instagram pic of Thelma and Louise over the cliff, etc.) In addition, I may not actually want to check the status of every build submitted, as that becomes noise after a while. I may want to know only when a build candidate has passed and is ready for promotion to a further testing environment. Since my workflow wrapper would be aware of this, it could automate the notification to me quite easily.

    Now add the consideration of management chain to this notification process. If I am out on vacation (which I expect to be able to do, since DevOps enables my personal life), my manager or my sponsoring executive may want to be notified if my application is not going to production for some reason during my absence. This is a level of automation that most build or deploy tools are simply not designed for. They have no idea who I report to and who that person reports to. But that departmental notification escalation chains can be incorporated easily into a workflow wrapper, and execute as expected while I am away (including pinging my cell phone just in case I need to jump in and fix something).

    Release Management Solutions

    Most release management (RM) or release orchestration (RO) tools have at least minimal workflow capabilities built into their offerings. Whether this capability is enough for your organization or whether you need a more capable workflow engine to wrap around it is a decision only you can make. Most release management tooling is designed to put an elegant wrapper around build automation and deployment automation. It usually contains the ability to segment by line of business or product, as well as role-based access controls (RBAC) for permissions and auditory compliance. And it usually contains at least elementary notification capabilities. What RM excels at are human interaction and tracking progress, and so it does meet the needs of many companies.

    When RM is not enough is usually when the DevOps services you provide begins to extend into ancillary services and secondary automation integration capabilities. Examples include monitoring systems integration, multiple testing tooling integration/evaluation, configuration management database (CMDB) integration and capacity-on-demand environment provisioning.  Highly evolved notification systems tend to require an external workflow wrapper beyond the normal capabilities an RM tool offers. A solid workflow wrapper acts as the glue that can hold disparate DevOps tooling (either from different vendors or performing different functions) together with further super-automation processes above them like an overarching umbrella.

    Impacts to the Bottom Line

    Choosing to offer DevOps services without any form of a workflow wrapper is like going hunting with an excellent rifle and no bullets: a lot of capability that can never reach the potential it was intended for. On the other hand, obsessing about replicating current processes within a workflow tool is like hunting a bunny rabbit with a bazooka: way too much damage caused when trying to keep things small and elegant. The key is to balance the use of a powerful tool (workflow), with Lean thinking in automation processes and constructs. Keep the building blocks small and functionally oriented; limit human interaction; and use workflow for the functions it is best-suited for—connection and communication.

    The technical capabilities a workflow wrapper will add to your DevOps solutions are substantial. Your services will be far more efficient and far more powerful. However, to truly assess the financial impact of a highly functioning workflow system, one must determine the value of time and decision-making. An elegant notification system can keep you informed without sending you countless emails you learn to ignore or flooding your text inbox with data that exceeds your cellular plan. In this way, notifications are meaningful, and you can begin to make meaningful business decisions based on the information you receive, only at the times you need to see it. What is that worth to you and your organization? Only you could really assess that.

    To continue the conversation, feel free to contact me.

     

  • DevOps: Commoditizing Infrastructure

    DevOps: Commoditizing Infrastructure

    Hardware vendors face a quandary. How do you distinguish yourself as superior from your competitors if you are unable to continue to grow a distinct architecture as your competitive advantage? But this dilemma does not begin and end with only hardware vendors; operating system vendors and virtual machine vendors are on the same horizon. In an effort to “uncomplicate” hardware idiosyncrasies, our industry evolves efforts from open standards to least-common-denominator functionality for a “common” language or set of features, and over time technologies such as containers emerge. If I can abstract my application from the “uniqueness” of hardware vendors, from the complexity of configuration settings, and make only assumptions about basics such as CPU, disk, memory, etc., then I am free to focus on the application and allow my “hardware mitigation technology” to perform its function.

    The Relationship Between App and Storage

    But to accomplish this feat, I must agree to a basic set of features or functions I intend to call, such as making “x” amount of machine memory available to my application. Whether disk or RAM, I am looking to define what my application needs and then expect the container (or hardware mitigation/abstraction manager of my choice) to manage the actual allocation. Then the quandary begins. Asking for 1TB of disk space as an example is generic enough. But does my request include the ability to specify I am looking only for disks that spin at more than 7,500 revolutions per minute, or will I be saddled with 5,000 rpm disks? There is a clear implication in speed for my app.

    Ultimately, as I sit and tune my code because my app has become “slow,” it could be something as innocuous as having its actual execution occurring in a cloud provider who uses the slower disk revolutions as opposed to the faster ones. Worse, it simply could be that the same cloud provider has a batch of the slower ones in one of its data centers where I wind up executing (only on a temporary basis), and I am never fully able to figure out why my app got “slow” for a while. That kind of random bad ju-ju requires the execution of a chicken to fix (I prefer Popeye’s, myself).

    Different storage vendors might have additional peculiarities that make one or the other superior in speed, reliability or performance over their respective competitors. But the container’s main functionality is to provide “disk” to my app. Specifications of specific disk vendors or providers are the problem of the cloud provider, not the app developer, in theory. But hardware refresh can be pricey, and to make a claim that “all our disks” have rpm specifications above 10,000 will be meaningful only for about 10 minutes when the 12,000 rpm models come out. And of course, there are many more architectural designs in the engineering that are intended to boost speed and reliability. If these are not to be ignored, a decision must be made as to “who” has the responsibility to manage their usage (usually the cloud provider), and “who” will complain when things get slow (usually you the customer).

    And what happens when “breakthroughs” occur in a specific technology? Solid-state hard drives, for example, are not exactly new now, but they once were. When you move your app from operating on a high-speed hard drive that spins to a chip-based hard drive the operates at the speed of the RAM it is based on, the difference in speed is an order of magnitude better. The inevitable question emerges, Will my container have the ability to specify RAM-based disks instead of traditional spinning disks, as part of the disk commodity it is providing to my app?

    Since the difference in speed is so great, you would think it would come up with a way. You would equally expect that providing offline storage such as tape (from the age of the dinosaur) would have its own designation in my container structure (not be equated with offline spinning drives due to the speed differential). When you go to make a backup or restore, the difference between tape and hard drive becomes as apparent as the difference between solid-state and spinning drives on app execution. So when a platform evolves, my container mitigation must evolve with it. But to offer the uniqueness of vendor superiority requires my programmers to learn who to request (or what level of hardware specs they must know), which is the whole point of wanting a commodity in the first place: I don’t have to know.

    Impact of Commodity on Other Components

    This same dilemma can be exhibited in CPU vendors. Intel touts its superiority, AMD would disagree, and there may always be an upstart who wants to rival the current leader in any space. Lest we think Intel-based computing is the undisputed champion, quantum computing may upset that assumption (unless, of course, Intel corners that market as well). The point here is simply to ask: how many features, how many specifications, how much do we the programming team need to know about our platform provider to understand why and where our application runs better and worse? In the days before container-style ideology, hardware vendors touted their differences and their uniqueness as the reason they were better. Now what?

    Then comes our virtual machine providers, and eventually our operating systems. If I want to truly abstract from my hardware layer, I must not call an OS-unique command that only Linux or only Windows can do. I must stick to functionality that both OS instances can reasonably understand and accommodate. While this least common denominator is great for extending where my app can run (on multiple types of hardware, in multiple types of OS installations), it has the same effect on uniqueness and potentially on competitive advantage. If the intent is to come up with a one-world OS, I am generally in favor, but our journey there is sure to have certain casualties in innovation that a single OS is suited for and its competitors are not.

    The inevitable question becomes, How far down, or how far up, do we want to form commodity provisioning of computing capabilities? One could argue that the simplicity of the iPhone platform is a user-centric expression of commodity provisioning at the top of the stack. Drag-and-drop programming languages are yet another expression of bucketing computing capabilities into more simplistic expressions (that more people can understand and, therefore, utilize). Commonality and simplicity do increase audience. In many segments of our industry, this is the nirvana of sales to the public.

    Impacts to the Bottom Line

    DevOps focuses on enabling innovation. At its core DevOps is focused on software innovation. Let’s face it—changes to hardware platforms are generally less frequent. If those hardware platforms are married to a particular app (not serving a large number of apps on the same box, same virtual instance, etc.) then at least provisioning can be tuned one to one with the app. But with the desire to turn infrastructure into a generic commodity that application programmers don’t have to think about comes the fact that they stop thinking about them. Over time they will understand less and less about why things become slow, or work better, as the knowledge of uniqueness goes away.

    Depending on your market focus, ubiquity is either your friend (as it touches consumers) or your enemy (if you are the hardware vendor attempting to maintain relevance over your competitors in an increasingly commoditized market). The questions I raise in this article are designed to focus on how to keep our trend toward hardware mitigation from stifling innovation in the very hardware platforms we depend upon. In any case, I welcome your thoughts on the topic.

    To continue the conversation, feel free to contact me.

  • Integrating ARA into traditional RM

    Integrating ARA into traditional RM

    Application Release Automation (ARA) is terminology that includes several DevOps constructs from a singular, application-centric point of view. In essence, ARA implies the automation of a single app from build through deployment (even if that deployment is to production). Of course, embedded or implied in that automation terminology is build automation or construction; deployment automation, which may include provisioning; and governance automation, which includes testing and approvals.

    If your company’s portfolio of applications is small, or if the applications are fully independent from each other, using ARA to describe your DevOps continuum is more or less a complete vernacular. However, if your application portfolio contains interdependencies; you need a coordinated/scheduled release event of multiple applications into production in a particular sequence or timing;or you are required to meet blanket compliance governance requirements that exceed the scope of a single app; you will need to learn how to integrate an ARA-based DevOps strategy with the more traditional discipline of Release Management (RM).

    Clarifying ARA

    One of the nicer implications of ARA thinking is that it places the value—or, accentuates the value—where it should be; that is, on the application. Automating infrastructure is great, but if no application rides on top of it, the infrastructure is essentially nothing more than unexplored capacity. Most CIOs are not fond of having excess computing capacity sitting idle. So the idea of ARA sets the focus of the automation where it should be: moving an “application” from construction to production release.

    ARA is also development methodology-neutral, which means the build, deployment, and releasing automation will look the same regardless of whether the code was prepared using Agile or Waterfall. Mind you, Agile tends to drive the need for speedier delivery systems, but even legacy development teams can find value in consistent delivery system automation once a build is ready to begin. It is, after all, consistency that reduces support costs by reducing errors and creating predictability, thereby improving quality. It also is consistency within the automation that permits scale, whereas variability in the automation deters scale.

    ARA also allows for depth within the stack. If code reigns at the top of the stack, then the operating system of the server reigns at the bottom (perhaps networking in the wine cellar and facilities management in the root cellar). ARA incorporates the notion of automating dependent layers of technology under the code such as database services, middleware or messaging services, perhaps the provisioning of a virtual machine, even down as far as configuring the OS, if needed.

    The practicality of moving down the stack however, is that the scope of change will continue to have a broader impact than a single app. I usually do not install a single instance of Oracle for each application I create; more often, I have several applications relying upon a single instance of my Oracle setup. So the actual changes I automate as I move down the stack tend to be more of a configuration adjustment. It is more likely my particular app needs a config change to accommodate it rather than a completely new install of a supporting technology component underneath my app. It is also worth noting that the frequency of change as you move down the stack slows down considerably compared with app code riding on it, and the cost or impact of change increases greatly, as you begin to impact a wider number of things riding on it.

    Deconstructing ARA

    ARA automation, then, includes all the traditional DevOps disciplines, but limiting them to the viewpoint of a single application as follows:

    Build Automation: This includes version-controlling each particular build. It also includes packaging all the component technologies to create a deployable package used for deployments throughout the software development life cycle (SDLC). The golden rule here is create a “build once, deploy many” philosophy in your build automation. Changes to code, included configuration files or dependent technology components down the stack must occur in the DEV class environment. Once the deployable package is completed to satisfaction in DEV, it will move unaltered through any other layers of testing environments before it gets to production for ultimate release.

    Deployment Automation: Since each deployable package is tightly version-controlled during its construction in build, there is now an ability to roll forward (normal deployment) or roll backward (revert to a prior version) in the event of failed testing in a given SDLC environment. Some of the common environment classes include integration testing, performance testing, user acceptance testing and pre-production quality assurance (QA) testing. Smart DevOps deployment tools utilize the ability to deploy (or overwrite) only the components that are out of sync with the versions listed within the deployable package. This smart deployment capability reduces time and ensures consistency with what “should” be there.

    Provisioning Automation: This is an offshoot of deployment automation. For those using virtual machine or cloud technologies, the deployment automation can include the ability to provision or set up an environment of any class on the fly. When testing is completed, the same environment can be decommissioned to not waste capacity or budgets. Provisioning bare-metal servers is precluded by human interactions, although what should go on them also can be fully automated once engineering is done with the physical work.

    Governance Automation: Even though substantial testing can and should occur in development, a significant amount of it must occur in other testing-class environments later in the SDLC progression. It is, therefore, a good idea to have the ability to isolate testing automation outside of both build and deployment templates or scripts, still making it callable or executable from an overall workflow manager (often included in the release orchestration or release automation tooling within DevOps). This gives the flexibility to run a series of tests without having to rebuild the application or redeploy the application just to kick it off. In addition to testing evaluation, there may be other process regulations driven by a governance team or compliance entity within the organization. All of these flags must be met and passed before final promotion to production.

    Integrating ARA with RM

    Dependency is the hobgoblin of simplicity. Applications that are fully independent of each other can utilize an ARA strategy and deploy into production as often as they want. No one else is impacted by this. But if a given application has dependencies upstream—I get my data from, or transact with, someone else—or downstream—my data is provided to someone else—then my path to production must include testing an application different from my own. Determining where the responsibility for failure lies now requires policy instead of just common sense. Synchronizing in-state changes where two dependent applications are now undergoing change at the same time requires policy to govern as well as more sophisticated SDLC testing environments to manage.

    When a wider view than just a single application is required, the traditional DevOps discipline of RM or its synonym release orchestration (RO) becomes the solution. RM allows for the sequencing of applications going into production to occur as needed. It allows for testing to occur when a dependency spiderweb is kicked off. RM allows for corporate governance to be applied universally to all applications attempting to enter production, instead of having to build it over and over again in each deployment script or template. RM can produce sophisticated logs designed to satisfy outside regulators for compliance history automation. These kinds of tools become invaluable when there must be a wider view of deployments to production beyond a single app at a time or in a small portfolio.

    Given this reality, the key to integrating ARA with RM is to ensure automation is constructed independently by discipline, and can be executed by discipline as needed. ARA encompasses or groups four disciplines together: build, deploy, provision and governance. RM uses the ARA package and extends it with other applications to create a needed sequence, or kicks off selective smaller units such as a testing package when needed to meet certain outside app change conditions based on dependencies. RM can selectively execute the automation ARA creates as it needs to to accommodate larger goals, while ARA continues to focus on keeping its own app as efficient as possible.

    Impacts to the Bottom Line

    There is no one right answer. There is only the perspective you hold within the DevOps continuum. Knowing that both ARA and RM are ultimately dependent on traditional DevOps disciplines offers flexibility on “how” you think about delivering innovation. In my opinion, ARA is simply the next evolution of automating build and deploy. It is a logical next step.

    There are other disciplines within DevOps, however, that might benefit from a wider view of the workflow required to move innovation to production. Take, for example, the integration of monitoring automation in the process, capacity on demand provisioning, security oversight and configuration management database (CMDB) integration—a wider view of how and where these services are integrated with DevOps might reduce the labor required to create them for each app, reinventing the wheel—or worrying the wheel has not been updated properly in a while. At its heart, DevOps attempts to create singular automations to perform tasks, increasing speed, reducing risks and improving quality.

    There is no immediate financial impact of this other than to pursue a strategy of single-app focus in an environment where a wider view would have been more ideal. Calculating the cost of that error is akin to picking six lotto numbers from the sky: It is always hard to know what might have been; at that point all one can do is wish.

    To continue the conversation, feel free to contact me.

  • Engaging Adoption of DevOps

    Engaging Adoption of DevOps

    Executive mandate does not immediately translate into a change in behavior. Organizations inherently grasp the value of embracing DevOps, but the timing of exactly when it begins and is completed is another matter. The devil truly is in the details—or rather, in the ability and willingness of the organization to adopt a change in thinking and a change in the actions that follow.

    “Adoption” is then a common term to describe the measurement of how far along an organization is in its transition between legacy and future, between the beginning and the completion of embracing DevOps services. As you might guess, adoption does not happen overnight, nor does it occur because of mandate or willingness. These factors influence the speed of adoption in some cases, but can act as deterrent in others. Adoption—like most other services provided to a given organization—occurs because its value proposition is sufficiently defined and resources are budgeted and assigned to it.

    The level of resources you dedicate to tracking and facilitating adoption of DevOps in your organization is directly proportionate to the complexity of your development efforts. If for example, you have a new development team well-versed in cloud technologies and comfortable with using software-as-a-service (SaaS) tooling with only few applications under development (small app portfolio), your proportionate adoption efforts will be small, and transition should occur very quickly. However, if you have a large number of applications spread out across a wide variety of hardware platforms and are using a wide variety of development languages or tools, your proportionate adoption efforts will be far more substantial to keep the timeline manageable. In either example, a complete lack of attention to adoption results in elongating the timeline two to eight times longer than it would have been if formally engaging adoption was considered.

    Adoption Engagement 101

    So what do these adoption/engagement resources do from day to day? The primary focus of the folks assigned to this duty breaks down into three major categories of work efforts:

    1. Communications conduit: The first responsibility of the folks on this team is to develop and implement a communications plan to keep all stakeholders up to date on what is going on with DevOps services. This includes metrics-based information such as how many applications have been fully onboarded to DevOps. It also may include how many applications have been onboarded into a particular level of automation within the stack, from code to OS. A company may want to know how many apps have gotten their database efforts automated, as an example, or perhaps how many apps are fully automated with middleware messaging systems. But the communications also must include keeping executives up to date on the ROI progress, and what areas of the company may be exhibiting resistance (for both legitimate and non-legitimate reasons). Expressing these outcomes can be sensitive and should be done with a modicum of care, so the folks assigned should have these skills to start with.
    2. Information Preparation & Follow-Up: When a given application in the portfolio is targeted for onboarding to DevOps, a significant amount of information is usually gathered about it. While a company can create an automated system for development engineers to provide the data into an electronic web page or other means, very often questions arise about “how” to respond. You do not want to have your DevOps services engineers constantly deterred from their work to answer basic questions such as this, so having an adoption team able to provide the data quickly facilitates the overall goals. In addition, often the development engineers are too “busy” to provide the DevOps team the data they need. Again, someone needs to follow-up to ensure questions are actually answered. While it is easy to assume engineers will behave professionally and respond accordingly, that assumption in the real world results in long delays and interrupted workflows across multiple teams.
    3. Training & Feature Expansions: Finally, the tools and architecture practices behind them need to be communicated effectively to the practitioners and participants of the DevOps services. When new programmers are hired, for example, they should have access to training materials on accessing to the DevOps toolsets and using base functionality, what governance procedures will impact them and more. On occasion, to facilitate adoption, an application executive sponsor and his or her immediate direct report engineering team leads will request a demonstration of a given DevOps tool or function. The adoption team should handle these requests and not divert a production DevOps engineer from his or her duties. And when a DevOps tool is updated or upgraded, the adoption team should develop the method to communicate the new ability, edit the existing documentation and ensure every practitioner is notified about the new features and how to use them.

    Adoption Engagement Financials

    Deciding how many people to dedicate to this function is a mix of your development environment complexity and what you have currently to complete your DevOps rollout efforts. If you want a 24×7 rollout, then you will need staffing levels to accommodate three shifts, seven days a week, with additional folks for vacations, holidays and sick leave. If you are only rolling out a 9-to-5, five-days-per-week effort, staffing numbers drop significantly. Resources, then, can be organized as one per line of business or one resource for every 50 applications in the portfolio (higher ratios are supportable). Whether you align the resources to the services you provide or to the customer audiences you support won’t matter nearly as much as having at least some resources dedicated to these efforts. As I stated earlier, a complete lack of resources dedicated to this function results in greatly extending the timelines for adoption of DevOps; worst case, in the failure of the transition entirely.

    This kind of function is inherently considered overhead. Most DevOps leaders (generally emerging from an engineering background) naturally don’t consider this more administrative function as necessary. It doesn’t strictly add to the engineering capability or skills of the main onboarding teams, so they simply do not include it or divert budget to more “necessary” folks during planning. Organizations that don’t actually provide a guided DevOps service—that is, they simply install the tools and tell the developers to have at it—find their adoption efforts lagging quite a bit as well. The early adopters jump right on without coaxing, but the folks too busy with their own efforts to “try something new” never seem to get around to it. Finally, the guys who think, “Since it ain’t broke, don’t fix it,” will never adopt. Even in this kind of tooling-only services deployment, dedicated adoption resources would have resolved these issues in a far more timely manner. The overhead may be painful, but the alternative is far more painful. Once a company or division experiences a failed DevOps rollout, it rarely returns to try again, and the overall costs could be doubled or worse.

    For organizations with the vision and perseverance to employ dedicated resources to facilitate adoption, the converse of the problems outlined above becomes the norm. Developer teams have the information they need to understand their own need and, more importantly, the benefits of adopting DevOps. The work relations between DevOps service provider and DevOps service consumer become highly efficient and mutually beneficial. The entire organization benefits from effective communications from the executive sponsors at the top to the engineering practitioners at the bottom. Execs know the costs, the progress, what is new and how it will help them reduce costs or speed up delivery. Practitioners know how to use the tooling, why it will help them and how to stay current on the changes that come along (and they invariably do). Practitioners appreciate being current on a given tool or skill set; this provides a way to do just that.

    Impacts to the Bottom Line

    Discreetly measuring the ROI for the application of adoption/engagement resources is a bit squishy. As a general estimation, take the timeline of the DevOps rollout you originally planned and multiply that by two to three. The cost of that delay in timeline (plus the risk of complete failure) is the cost delta to measure against hiring a few more folks (or allocating them from lesser-used areas already on staff). Usually the cost of failure far outweighs the benefits of expanding the overhead support teams as needed. In addition, the cost of a failed communication effort can grow over time. At the outset or launch of DevOps, the communications, whether in document or oral forms, will be fully up to date. But over time, as vendors add new features and new versions of the tooling emerge, practitioners won’t know how to take advantage of the new capabilities. Since many of these new features enhance speed or accuracy, the tendency to not use them—or use the inconsistently for lack of knowledge—will add up.

    I find the resistance to the idea of adoption or engagement resources often is founded in the notion that engineers are plenty smart enough to figure this stuff out on their own and they will naturally gravitate to it, as ultimately it makes their lives easier and better. Yes, the engineers are smart enough and some will. However, engineers often build their own “ecosystems” of how an application should be built (which few others could understand or improve upon). They often are reluctant to tear this down in favor of corporately implemented templates (that everyone could understand and improve). They fear the loss of unique skills the company must depend upon. They fear the loss of being “special” and needed. Even though DevOps standardization makes builds and deployments consistent, reduces trouble tickets and improves quality and speed—which ultimately translates into fewer 2 a.m. phone calls for engineers to come fix—they are still reluctant to change (just like the rest of us). Having dedicated resources to help them through the value of this, and assure them of their continuing value in a differing area (the code we want them to write), can make a world of difference and ensure a transition does occur.

    To continue the conversation, feel free to contact me.

     

  • Baselining Metrics to Measure DevOps ROI

    Baselining Metrics to Measure DevOps ROI

    Assessing the ROI of DevOps normally is done against a company’s baseline cost. The problem is, most organizations do not collect or evaluate the cost of “automating and delivering” innovation; they usually are more focused on the innovation product itself. As an example, if a company creates an application hosted on the web that offers its customers the ability to purchase inventory, the company usually is obsessed with knowing how that application performs: How many people are using it? How often does it fail due to bugs? How many sales are made? The company typically focuses on the end product—on the application as it touches customers and potential customers. The delivery process that precedes the application being loaded into production is not usually the subject of intense scrutiny. At least, most organizations are not evaluating the speed of delivery and process effectiveness of delivery prior to an embrace of DevOps methods and tools.

    This lack of scrutiny results in the lack of a baseline to compare against. Say, for example, a typical programmer (prior to DevOps) uses a mix of scripts, human intelligence and human verification to create and verify a build. Prior to DevOps, the goal is simply to create the build and get it out there to look at it, and decide what to do next. Most programmers are not obsessed with how efficient this entire process is; for them, it is a means to an end. So most programmers do not keep records of the average amount of time it takes to create the build from beginning to end. This is even more true of the deployment process. Once a build has been created (again prior to DevOps), getting it deployed to the target environment might involve a few different teams (depending on the class of test environment that is targeted). The process may involve email requests to other support teams or filling out a ticket to get it done. Once again, most organizations spend little time tracking this process from end to end; it is only ever a means to an end.

    If, in fact, an organization was watching closely how long it takes to get a build created and how long it takes to get a deployment completed, that organization would be running toward DevOps at full speed to see improvements it otherwise could not achieve. It is the things we do not measure that receive little attention—until they break. Once our deployment process has become so onerous or so complicated or so burdened with approvals required that nearly nothing gets deployed, then we begin to look at it. This creates a culture of “squeaky wheel syndrome,” which makes both our development and our operations teams focus only on reactionary issues, while proactive concerns die on the side of the road. There is no time to invent proactive improvements when you are constantly fighting the fires of something else that is seriously broken. This culture must also change if DevOps is to succeed broadly in your organization, so keep note of how far it may be infected in how you run things today.

    Preparing for Collection

    Given that most companies lack a real baseline to compare against in this area, realize that collecting one will take an investment itself. This baseline metrics collection cost is usually added to the cost of the investment in DevOps. It is a bit unfair, as you are effectively saying, the soap company will also have to bear the cost of determining how dirty we are before we invest in the soap that will clean us. The makers of soap tend to think you already have a sense of how dirty you are, or you would not be looking at buying soap in the first place. But without an accurate assessment of knowing the extent of the problems (in this case, the investment of time you spend today), you will never know the ROI of having made the improvement of DevOps (the after picture).

    Also, keep in mind that measuring anything manual is difficult. This sounds like a saying from Captain Obvious, but consider that manual work efforts require additional manual work efforts to capture them and record them in some kind of system. If, as an example, my typical programmer spends 15 minutes of every build doing human verification of the build, this amount of time is reflected nowhere. Hourly employees typically record how many hours they work per day, not what tasks they spent during the day. Even the employees who log work hours more discreetly tend to record only that they spent “x” amount of hours doing “development,” not what general task categories occurred within the heading “development.” So there are no electronic records out there that can be gathered automatically that will reflect the 15 minutes spent doing human verification of a generic build of an application. The only way to know this is a personal interview, followed by someone keying the data into a collection system of some sort.

    Knowing that the information you seek for a baseline will come in human interactions as well as some automated logs, start the collection using a tool that allows for maximum flexibility for those efforts. Spreadsheets come to mind. When the effort to collect a baseline begins, you will have some ideas about the kinds of data you wish to collect and how it might exist in your organization. But it is impossible to know exactly how the resulting efforts will be completed. In short, you don’t know what you don’t know. While you collect metrics you will invariably come across someone who reveals critical metrics-related information you were not “planning” to collect. But once the revelation is made, you will need to incorporate the “new” information to really provide an accurate baseline assessment.

    Determining the Baseline

    The next step is to begin the collection process itself. The critical data to examine usually can be gathered by looking at “human” time involved in a given core “process” of the automation and delivery of innovation, and then the “machine” time involved as well. For example, the time a programmer must spend prior to the launch of a build (not the coding time)—all the prep work doubling checking that the system is ready to launch the build scripts (or ready to manually push the buttons)—would be considered “human” time. The execution of the scripts to create the build would be considered “machine” time. Then, all the work done after the build is created to verify it and make sure things are as the programmer expected, again can be put into the “human” time category. DevOps will systemically automate both the pre- and post-time activities the “human” currently invests, so all that time will count in the ROI.

    The above example is good when one person has full control of a process from end to end. When other teams must get involved to complete a task, the “human” time begins to skyrocket. If, for example, a programmer or release manager, must get a request email (or ticket system equivalent) to deploy a particular build to a particular target environment, the entire life cycle of that ticket becomes part of the “human” time you need to track as your baseline metric. Even when shops create a two-day service level agreement (SLA) for turnaround, it still represents two days of delay (due to “process” time) in the life cycle of a deployment. The time “humans” actually spend reading requests, approving requests and routing requests is all time that DevOps will automate and eliminate. Finally, when the correct support team (usually Ops) pushes their own script buttons to do the deploy, that part of this process is once again “machine” time. As you can imagine, the end-to-end time of a given deployment can be days in the pre-DevOps methods and only minutes in the post-DevOps methods. Self-service deployments can reduce this time even farther, removing handoffs, reducing risks and delivering innovation at will.

    Further classification of this metric data can be useful as well. For example, you may wish to break down “machine” time into its relevant subcategories such as build script execution, deploy script execution, test script execution and release script execution. You also may want to break down the “human” time a bit further as well into pre-build/deploy verification, post-build/deploy verification and test script results evaluation. Finally, you may wish to break “process” time down further into request generation, request routing, request approval, request declination and request notification. DevOps will be able to give you discreet visibility into the time you saved in these subcategories, although since baselining is generally a one-and-done effort, you may not want to hunt this level of detail. It is down the road where this kind of subclassification becomes useful—for example when you are trying to determine if one set of tests runs faster than another or one cloud provider runs faster than another.

    Evaluating the Baseline

    Having gone through this exercise, a hidden or secondary benefit emerges. Your organization begins to understand much better what it has been investing in the current delivery of innovation processes. Generally, immediate areas of concern emerge and provide the DevOps services delivery with a road map for which areas to attack first to provide the company with the biggest bang for the buck. Without a baseline effort such as this in place, your DevOps services delivery plan will be based on anecdotal thinking or on executive mandate that may not be substantiated by the data on the floor. Having a baseline provides a road map for delivery based on concrete evidence on what should matter most to the business in making improvements. No one can argue with the numbers (assuming they were collected with care). The road map may reveal which applications in your organizations portfolio should be targeted first for DevOps.

    Also, your organization now understands much better “what” data in this process it sees as relevant to continue collecting even after DevOps is implemented. As DevOps is primarily an automation set of tools and methods, DevOps should be able to deliver these metrics in an ongoing basis. This allows continued metrics efforts to be automated without further human labor and can provide continuing proof of the value of DevOps to the organization, as over time the numbers continue to improve. Since the lion’s share of manual tasks disappear with DevOps in place, there should be no reason to have to manually collect metrics data as you move forward.

    Finally, having an automated metrics collection system put in place by DevOps, should lead to the next evolution in thinking: simulation. Well after the initial ROI for DevOps services has received the accolades it is due, an automated metrics collection system should provide all the raw data needed for some predictive analysis. Tweaking variables or process points or “machine” effectiveness should result in increased speed of delivery that can be measured (even before it is reality) based on the data you have been collecting systemically. Understanding the impacts of these changes is worth more than its weight in gold, because you can weed out the planned changes that simulation advises against before they even hit the road map, let alone occur in your world.

    Impacts to the Bottom Line

    There is already enough anecdotal evidence in existence to entice companies to embrace the DevOps journey. However, as your strategy begins to look to deepen down the stack from application code level to middleware, database and operating system, each layer could be assessed against a baseline to determine if your road map is sound and, more importantly, in the right order for your best ROI. This applies as well as you look to expand your DevOps journey laterally with additional services such as automated testing and evaluation, gated progression, automated provisioning, release orchestration, configuration management database (CMDB) integration, production provisioning on demand and monitoring integration. Each new service can be assessed against a baseline to determine if its ROI still meets the demands of your finance department and if your road map here is in the right order for biggest bang for the buck.

    The subtle win by performing baselining is the culture change it dispenses through your entire organization. Baseline metrics tell your development team and your operations teams (as well as the business units looking on) that you intend to measure performance and improve it systemically. Just because something was not examined before does not mean it will remain so, and a forest fire does not have to be the motivation to do so. DevOps is a proactive improvement to your delivery of automation. Collecting that baseline for your ROI, begins to seed the idea that proactive improvements are important to your executive teams. It is a demonstration of action over platitudes. It begins to eliminate the forest-fire responses in your employees, and creates a new forward thinking point of view. This cultural change is worth more to your company than any implementation of any one given tool or method, it goes back to changing how you think about delivering innovation in your individual job. That is what DevOps can do best.

    To continue the conversation, feel free to contact me …

  • Training & Certification Implications

    Training & Certification Implications

    Changing culture does not happen by osmosis, it happens through incessant education. The DevOps continuum of services your organization originally intends to create and implement is a journey more than a concrete destination. It will evolve over time no matter what your original plans were, no matter what your vendor of choice tells you, and no matter what resistance you meet along the way. If anything, resistance within your organization will push your journey left or right from its originally intended path. The lack of features in your tooling, or the introduction of those same features over time, will impact the journey of DevOps your organization will embrace. Culture is fluid. DevOps is fluid. So managing the impact they will have on each other should warrant dedicated concentration, and resources, in the form of training and education.

    Audience Educational Considerations

    It is not just your practitioner base that will need perpetual education to use your DevOps services to maximum effect. It is also the business community that relies on DevOps information and the executive community that may have sponsored the journey into DevOps in the first place. Education for the non-practitioner is more than just a matter of managing expectations, although that is a big part of it; it is a matter of helping this community learn to watch “change” or innovation as it moves from idea to production. DevOps will make an extraordinary impact on the speed of innovation delivery, as well as the quality or reliability of delivery. Engaging your business community and training them so they see this, comprehends what it means to them, and keeps them informed of the improvements will be critical to the long-term success of your efforts.

    Consider how managing expectations will impact adoption of DevOps services from a business point of view. When you first start on your DevOps journey, usually only the build and deployment services are launched as the “core” of the service. As you do this, you will need a way to keep your business community (i.e., the audience of what you do), informed as to your progress, and what key features or functions remain on the drawing board. How you communicate this information will be key to managing the expectations of your audience.

    For example, your initial efforts at build and deploy may only include the application code as its target. But over time you may begin to deepen what can be automated down the stack—say databases or middleware—as your next target. Eventually you are automating server O/S or even network definitions. If this is the services plan your organization takes on, you need to outline the timeline for expansion to the business community—not just in bullet points on a chart or timeline graphic, but specific benefits they should expect to receive at each milestone (i.e., why they should care).

    As your actual journey of DevOps begins, you may find, as an example, that you never get to network definitions in your automation efforts for one reason or another. Why you’re not reaching this goal needs to be explained, as well as the impacts of this. What drives this conclusion should also be explained; it might be for a very good or legitimate reason. But altering the DevOps journey needs to be ingrained in the business culture. Taking small steps, minimizing risk, learning from mistakes, and moving on—all are key lean concepts, agile methods, and ultimately they ensure the success of your DevOps efforts. The need to communicate changes in your plans to the business community helps manage expectations, but beyond that, it teaches the business community that information will be flowing constantly—and will be important constantly, even if it is brief in each segment.

    Whether your organization plans to expand DevOps services by moving down the stack, or perhaps by adding more capabilities laterally, you will need the ability communicate direction as well as “how” to take advantage of the expansion of DevOps you offer. So for example, if you are expanding your DevOps service by moving down the stack from app to operating system (OS), you will need to show the business community how to watch changes at each level (app, middleware, OS) or, if you are expanding your services laterally, adding automated testing, then release orchestration, then monitoring integration, and finally production provisioning automation. You need a plan for communicating each success, as well as “how” the business community can take advantage of what is accomplished.

    This communication plan should manage expectations, inform of changes in strategy or direction, inform of success/failure/alteration, and teach the business user how to use what you have done. Your communications plan may be looking at new data that was not previously available, or tracking innovation change differently. But educating your business community base will not be a “one-time” event; it will be a perpetual activity. Therefore, how you create this process should be built for perpetual delivery, not just a one-and-done delivery.

    The frequency of offering it will depend on the specifics of your organization’s needs. But the frequency should always keep pace with the changes and successes of your DevOps services delivery. If that is weekly, then weekly, if that is monthly then monthly. What you do to communicate, and how, will depend partly on that frequency and the size of your audience. Keep in mind your tooling vendor is also constantly enhancing its toolset with new features. The frequency of delivery may also need to incorporate those impacts as well.

    Practitioner Educational Considerations

    Educating your practitioner community generally is focused on tooling proficiency. But that may be a limited view. The architecture of your DevOps services is arguably more important than the specifics of how you use the build tool. Consider how your DevOps practitioners intend to accomplish the ability to scale build automation, as an example. I could just train the practitioners on how to use the build tool and then turn them loose, automating builds armed with little more than a cursory understanding of my preferred build tool. And over time using this training method, I will have as many methods to construct build automation as I have engineers who touch my tools, namely “n” methods. Having that much diversity in “how” I construct build automation creates variability that, with a large portfolio of apps, does not scale and is difficult to sustain without constantly throwing labor at it.

    Conversely, my architectural approach to build automation established by the DevOps team instead might be that each kind of technology uses a master template, or framework, as its approach. This information has little to do with a specific build tool and everything to do with the  organization’s “strategy” for build construction. That information becomes much more important than specific tool usage, because it affects “how” engineers use the tool according to company standards and practices. Training and creating a communications plan for reaching every member of your practitioner community on these items becomes critical to longer term success. This includes hiring new talent, departmental transfers and contractor augmentation.

    The additional consideration here is how you want to manage your intellectual property. “How” your company constructs build automation may be a proprietary advantage for your firm. This is particularly true if your architecture team is top-notch and able to create structures in your tooling that less experienced practitioners would not have even considered. If you train outside contractors on your methods, you are in effect giving away some of the farm. Non-disclosure agreements may keep a contractor from purposely sharing your methods with a competitor, but it will not prevent the engineer from learning how to make a better mouse trap while working with you. The contractor leaves your employ, a much more well-educated engineer than when he or she started.

    Once overall strategy education has been planned for and imparted to your practitioner community (another perpetual delivery method required), most companies focus on tooling proficiency. DevOps is still a relatively new phenomenon. And as such, certifications that are common in the networking world, for example, do not exist in the DevOps world. Gaining proficiency from vendor tools should be provided by those vendors as part of the licensing purchase agreement. If the vendor is unable to provide it in a formal delivery method, beware: The toolset is probably a new acquisition or too new in the market. If you are using open source tooling, a fully in-house model for training must be created and managed on an ongoing basis.

    Impacts to the Bottom Line

    Changing culture on a wide scale will require a focus of resources for training and education across your company. Engineers typically do not make the best communicators—that might be a DNA thing—so you may want to pair training and communication resource employees with the DevOps team to deliver what will be needed in an ongoing basis. The main consideration here is to realize that because DevOps services constantly will be evolving, communication plans need to keep pace with their evolution. And yes, I realize, this is additional overhead costs that almost no DevOps delivery person believes he or she needs when the services are first launched.

    But like all things, an initial publication or announcement is not sufficient to keep track with something that changes on a highly frequent basis. Adoption either can be facilitated by a good communications plan or thwarted by the lack of one. If it takes your company 15 months longer to get fully onboarded to DevOps because of delays in the rate of adoption (based on poor or non-existent communication), you must consider the costs of that delay in contrast. If after onboarding to DevOps, your business audience is ill-informed about how to use the services to maximum effect, what will that inefficiency cost the company? If your practitioners are only exposed to tooling proficiency once and without architectural oversight, what will the cost of the variability they produce in their automation be to the company?

    Investing in training and constant communication may not raise revenue, but it will certainly avoid costs. The measure of how much cost it avoids is difficult to track. Most companies who make the investment understand intrinsically what the alternative would be, and choose not to go down that road. The only admonition I would offer is that I have seen clients make this decision late in the cycle, and every one of them did so because the pain motivated it. Your strategies for DevOps should be comprehensive in perspective, if you need help crafting or improving yours, contact me—I am looking at the moment.

    To continue the conversation, feel free to contact me.

     

  • Mobility and Quantum Adaptation

    Mobility and Quantum Adaptation

    As you look at the preponderance of DevOps tooling solutions available today, you notice similarities in deployment automation and how they work. DevOps is purported to be the most significant catalyst for change in how we innovate in the last two decades. Yet, as you examine the tooling and the strengths and weaknesses of each, so many products are strikingly the same. It seems to me that if DevOps is to ingrain itself in IT culture, it must use its own principles to remain current, or remain fresh in the coming years. The doctors of IT must learn to use the tool of change to change it once again, and then again, and so on.

    Case in point: The deployment discipline of DevOps is premised largely on a “controlled” rollout. Deploying versions of code and versions of applications into production is a fully controlled event. Our entire release orchestration (RO) discipline is merely an extension of our basic deployment tooling. RO tools add governance and quality assurance (QA), segregating it directly from a specific deployment and handling it more broadly in a release event. But whether we are examining a deployment tool or release orchestration software, both rely upon a premise of controlled distribution of applications to their respective destination.

    The Mobility Disruption

    Mobility, or mobile apps, disrupt this thinking. The platform our Apple partners have given us (in its current form, though I understand that is evolving as we speak), is based on an “at will” acceptance of a new version of a given application. When I look at my phone, I see a red balloon informing me of how many application “updates” are available for me to accept. I can decline an upgrade just by not taking any action. This is true, even of the newest version of the Apple OS (the foundational building block of application capabilities). Or, when I am home on my Wi-Fi network, I can at my leisure begin downloading and updating the various software application versions to the latest offerings. All are my actions, determined completely by my choice.

    It’s a little different on my PC—I have more flexibility of what version I have running on this platform, at least in terms of rollback. If I do not like the next version of my web browser, I can pull up the site and choose an older version to use on my PC, until the errors that bothered me are resolved in the latest offering. Mobile apps, in the mobile catalog so many of us use on our phones, offer no such options. Once I have updated the application to the latest offering, I am stuck with it, errors and all. If it becomes too intrusive to my work efforts, I would have to pick something else to take its place, or just stop using it and wait until a new version is available again.

    DevOps deployment tooling is foreign to this model. The servers in production do not get a choice about whether they intend to accept our deployments, and when they might get around to it. They do what we tell them to do, and when we tell them to do it. But it seems updating our servers, upon which future versions of applications will rely, does not cover the last mile of our application delivery. Mobility is accomplished ad hoc, by consent, and without controls on my part.

    This has an immediate impact to my thinking about application distribution (and how DevOps will play in this new world). If I make a catastrophic change to the servers my mobile apps rely upon (that is, a change where only the new version of the app works), it will require my users to upgrade, or stop using the product as it sits today. This is dangerous if my application is utilitarian (not critical to daily life), and could be replaced by a competitor. It might mean that my applications must seek out version-specific back ends if I want to have multiple versions still working until people get around to upgrading. That costs time, money and support. Either case forces me to think differently about distribution.

    You can imagine how angry you might be if your online banking software just stopped working. Mobility in its current form also makes rolling back to a previous version a near impossibility. So if the new version of the app has a bug in it that prevents key functionality, I am going to get a huge black eye with my customers, and must race to produce a new version minus the old bug, even assuming universal adoption. Alternatively, my customers may be using the buggy version for a while before they choose to upgrade to the bug-free version.

    There is tooling for corporations to more tightly control a provided device or even govern what happens to personal devices for corporate software. But the ability for DevOps deployment solutions to integrate with a mobility platform requires a new way of thinking. The limitations and the new capabilities must be taken into account specifically before any vendor can legitimately offer a holistic solution for DevOps for mobile.

    The Quantum Obliteration

    Quantum computing is likely to prove once again to be another disruption to our DevOps model. Evolution will help us bridge the gaps and minimize the impacts. But consider that quantum systems are not based on a foundational binary model. This is not like English speakers having to learn the Cyrillic version of the alphabet to read and write in Russian. No, it is more like communication will be telepathic from now on. The entire constructs of binary will eventually fade into the history books, and a new construct will emerge in its place.

    Today we use binary-based computing systems to attempt to converse and control the quantum systems built so far. When you don’t know how to communicate telepathically, you use what you do know: reading and writing. But it will take a long time before the power of quantum computing is truly unlocked; until then, we have to constrict its capabilities with binary-based computing methods. When native quantum languages, and perhaps hardware, are let loose, the true power of the platform will be better understood and impacts experienced.

    How will DevOps play in a quantum computing environment? Will the fundamentals about deployment and distribution once again evolve to incorporate the impacts of new quantum platform capabilities such as entanglement? Or will we experience what we have in the mobile platforms of today: a series of catchup steps to deal with what is different? If we use our catalyst of change in DevOps to begin to think differently about our own future and the capabilities our tools should provide, perhaps DevOps will still be unlocking possibilities none of us dream of today. If not, I foresee DevOps becoming only another fad, replaced by the term we give to the next wave of innovation thought needed to unlock our future.

    To continue the conversation, feel free to contact me.