devopsy.kinsta.cloud

Tag: architecture

  • All Day DevOps: DevOps at Massive Scale

    All Day DevOps: DevOps at Massive Scale

    When you have a billion users, people notice. That’s where our story about DevOps and Yahoo! starts. For Kishore Jalleda and Gopal Mor, both engineers at Yahoo!, when something goes wrong on a Yahoo! page, people will notice. Correction: a lot of people will notice.

    Of course, Yahoo!, like all services on the internet, constantly improves its products. In fact, it has 100+ iterations and experiments happening at any given time. Some changes bring new innovation to the forefront and others alter the user experience.

    When iterations and experiments are served in front of loyal users who have become comfortable with a specific user experience, they sometimes react with a natural resistance to the change. When change causes, or appears to cause, breaks in service, the backlash can be crippling. Frequent spurts of backlash resulting from these changes became known to some Yahoo! Insiders as “the bad years.”

    At the recent All Day DevOps conference, Kishore and Gopal shared how Yahoo! turned to DevOps practices to recover from “the bad years.” In their presentation, “Launching Products at Massive Scale: The DevOps Way,” Kishore said, “DevOps is about eliminating technical, process and cultural barriers between idea and execution, using software.” Specific to each of these, Kishore recommends the following:

    Culture of ownership and excellence: Own life cycles within Development, fix root causes and have pride in the product.

    Processes: Design or engineer processes to be fast and agile. Work should be iterative, support learning and provide fast feedback cycles. Let the machines to the heavy lifting.

    Tools: Solve operations problems with software. Use open-source tools that are self-service, reusable, friendly and easy to use.

    At Yahoo!, the DevOps practice is built on three functional pillars: deliver products to market quickly; prevent defects from reaching customers; and, repair production issues quickly.

    Speaking of repairing production issues quickly, Gopal discussed resiliency. With its user base, downtime (or lack thereof) is critical to Yahoo! and the challenges are many:

    • Distributed multilayer architecture
    • 100s of subsystems
    • Complex request flow
    • Change is the only constant

    While it may be counterintuitive, Gopal demonstrated how the combined system is weaker than the weakest subsystem.

    To ensure optimum uptime, Gopal said we need to:

    1. Analyze the entire range of failure types
    2. Understand their rate and impact level
    3. Plan to cover all failure types
    4. Conduct fire drills – test, test and test.

    Specifically, how does Yahoo! ensure high availability? It maintains four layers of resiliency in the serving stack:

    Speculative retry – Deliver the page again after a predefined latency is exceeded. This addresses longtail latency and intermittent failures.

    Per-module fallback – Cache non-personalized modules to front-end servers and serve cached content for failed modules. You need to ensure the cache is refreshed often, that you implement strong validation of the cache’s data and you check for backward compatibility if the TTL is high.

    Fullpage failsafe – Cache the entire page without personalized data and ads and with minimal interaction. This is used when the entire page cannot be served. Yahoo! uses auto-scale AWS servers to serve these pages so its servers are not negatively affected.

    Failwhale page –  The “we will be right back” page that lets users know, yes, your internet is working, but our service isn’t – temporarily. Please try again. Obviously, a last resort, but a necessary one.

    There is a lot here and you don’t have to have a service at the scale of Yahoo! to benefit for the experience. You can dive into Kishore and Gopal’s full All Day DevOps conference session (just 30 minutes) to learn more about their learnings from the DevOps front lines. The other 56 presentations from the All Day DevOps Conference are also available online, free-of-charge here.

    This blog series is reviewing sessions from the All Day DevOps conference from November, which hosted over 13,500 registered attendees. Last week I discussed, “Docker, the New Ordinary.” Next week, look for “System Hardening Using Ansible,” from Akash Mahajan.

    — Derek E. Weeks

  • DevOps and Automation Abstraction?

    DevOps and Automation Abstraction?

    Similarity breeds consistency, giving way to predictability and, thereby, dropping costs of operations and quality from a lower number of incidents and a shorter duration to resolve them. But at the heart of this chain of events is the recognition that we tend to do certain tasks the same way, over and over again, for a given type of technology. Variability then becomes our enemy.  Consistency becomes our ally.  So the question: How do we as DevOps service providers instill a level of consistency in what we do that is not engineer-specific (i.e. only one guy knows how it is supposed to work) and will still scale at the enterprise level?

    From a people perspective, we use teams, whether co-located in person or virtually using the same backbone of tooling over wide geographical separation. From a process perspective, we use governance and education to ensure the same processes and approaches are taken at a fundamental level across different types of technologies. And finally, from a tooling perspective, we use templates or frameworks (or pick your synonym) that allows us to construct a shell that guides the automation construction to be consistent from parent to child. The combination of these three fundamental elements leads to consistency that scales. But is that all?

    What Are We Missing?

    Could there be another fundamental element we have overlooked—namely, abstraction? Case in point: Consider the effect of a drag-and-drop interface versus a typed one to accomplish a given task. Both look to accomplish the same outcome. One could argue the typed text approach offers more flexibility and customization to get the task done. But the drag-and-drop method offers a consistent, simplistic way of doing a similar task, the same way, over and over again. Examining both methods objectively, the drag-and-drop method offers a level of consistency the typed method never could. For the typed one, you would always have to check to ensure punctuation and lack of changes (additions or deletions) to “know” it was the same script as last time. The drag-and-drop method on the other hand, is “canned” or largely unalterable; it may limit customization (by design), but it excels at consistency.

    The drag-and-drop method achieves this consistency by effectively abstracting the typed or scripted code within it from the user who employs it. The user only makes the choice to use it, when desired (or needed); the rest of the mechanics are abstracted from the user. The reason drag-and-drop methods exist is beyond just consistency, however; it is for simplicity. Apple computers would appear to be founded on this idea. For those of you old enough to remember DOS on PC computers, the parallel here immediately becomes clear. DOS was a typed-based command-line operating system (OS) fully capable of operating a PC many years ago. But Apple’s graphical interface made the same functions “intuitive” and forced the PC industry into Windows to stay competitive. Nobody chooses a command line OS like DOS when something as elegant as iOS exists, or Windows (or pick your graphical poison here).

    A graphical OS abstracts core functions from the user, and makes every attempt to make an elegant intuitive packaging of the same functions. Users “get” an iPhone, which accounts for its sales. You don’t need formal education classes or an eidetic memory to use an iPhone. Users effectively have been abstracted from all the typed code it takes to create an OS like this, millions of lines of such code packaged up into swipes, beeps and elegant graphical displays. Keep in mind, at its core this OS abstracts users from variability that might exist if hundreds of companies created their own versions of iOS. The simplicity is paired with consistency, and the quality of Apple is a result (for better or worse). But whatever the quality of Apple is, it is far better than if a wide degree of variability were permitted in the vendors of iOS or the construction of it.

    Extending Abstraction in DevOps

    Now enter DevOps. Containers are a level of abstraction widely popular in our discipline for keeping hardware particulars away from what are typically software engineers. The value of this abstraction is well-understood. My challenge is simply to extend this thinking into the next tier of DevOps tooling. What if the ability to include database DevOps automation was simply a product of drag-and-drop methods that every build tool understood perfectly? Granted, you may have to limit the expression of this idea to say Oracle or SQL Server at first to capture the nuances and particular capabilities of products and technologies in this space. But if someone built a “container” for Oracle, I would wager the choice to use Oracle would rise substantially over its competitors, at least until those containers or abstraction layers were just as effective with other database products.

    The abstraction phenomenon need not be limited to hardware, or as I have proposed to database platforms. It also could apply to .NET or Java development efforts at the top of the stack. The steps needed to take an idea, code it and put it through the build process are going to be largely similar when working in a .NET shop or a java shop (or pick your particular development tool of choice here). Even the top of the stack—that is to say, the development of applications itself—could be a viable candidate for abstraction from the mundane details as it relates to a build construction or a deployment to a given class of environment.

    Creating effective containers for the application layer in effect limits variability, and increases predictability. I realize it has to be technology-type sensitive. The .NET shops have their own container. The java shops have theirs, and so on. The only question is whether we are able to include enough of the common core steps we take to produce a build with a given technology type to form an abstraction construct of value. Over time, this would include every type of technology widely in use today.

    The effort to simplify through abstractions ultimately would result in our ability to include technologies such as WebSphere at our pleasure. It would also allow us to switch out WebSphere for a new competitor using the same abstraction constructs, universally, just by rebuilding all our apps one time. This kind of power is alluring to development teams. It drives application developers and operations engineers to begin to work together to look for the most cost effective technology that can perform a given spec. Being the best at something is no longer a cage from which there is no escape later.

    Technology components would be selected based on performance and cost of ownership (i.e. operations) later. Switching is ultrasimplistic if all we ever used to include it in our build routine was a drag-and-drop style method with universal, well-defined parameters. A customer’s ability to switch products to the best of breed through easy or simplistic means keeps pressure on vendors to actually be the best of breed. If the performance suffers or a new release is full of bugs, developers just switch it out through abstraction (not long, complicated scripts where everything has to go right, including a full moon on the third Tuesday of the month with a Gregorian chant in the background).

    Impacts to the Bottom Line

    The selection of DevOps tooling often is a cantankerous one in large organizations. Debates ensue about flexibility, extensibility and whether a given tool can be customized to accommodate the unique little snowflake of my development teams. Perhaps these debates are headed in the wrong direction. Perhaps instead of prizing and valuing the extensibility or customization a given tool affords, we should be looking closer at how that extensibility is achieved.

    For an enterprise to make an unwitting choice to get into the script business, is effectively stating: “I will be maintaining the scripts I use to construct a build with as much rigor as I maintain the code these scripts are designed to create builds out of.” So, we just doubled our programming staff without realizing it—one half working on the application code and the other half working on the scripting code that “automates” the delivery of a build during construction. While overall speed may be increased, it comes at the cost of doubling already expensive headcount.

    As an alternative, what if enterprise customers were asking tooling vendors how simplistic their tools are to use, and how standardized are their approaches. While we may not enjoy losing a degree of flexibility as we may be unable to customize a drag-and-drop method construct, the payoffs to the business are simplicity of use and scale at an enterprise level. Not only do I refrain from doubling headcount, I can actually free resources to focus on the disciplines and areas they enjoy most instead of performing script maintenance. Abstraction as a construct: From the hardware layer up through the application layer, it may be a driving fundamental that keeps DevOps scaling and lowering the cost of innovation for many years to come—but only if you begin to ask for it.

    To continue the conversation, feel free to contact me.

  • Dying for Intelligent Calendar Awareness

    Dying for Intelligent Calendar Awareness

    What would we die for in an integrated DevOps tooling solution—something that would make our operations partners stand up and cheer? Intelligent calendar awareness.

    The most fundamental construct of any release orchestrations (RO) tooling software is the calendar. RO software, no matter who the vendor is, uses a form of the calendar, to outline what tasks will get performed in what order from beginning to end. Better RO software allows for parallel tasking, but all RO software uses a calendar. So the most fundamental use of a calendar construct is focused on “when” tasks are performed and completed. Adding intelligence to a calendar construct adds three more dimensions or characteristics to this baseline feature.

    The Intelligence of Ownership

    First is the concept of ownership. Instead of just marking a calendar with the dates of specific tasks, or clusters of tasks, the addition of ownership marks the calendar with “events” that are “owned” by specific teams of people intended to perform groups of actions or tasks. As an example, instead of just marking the calendar that on April 27 this year, with the specific tasks we need to run a mini-release to upgrade our Oracle database from version “x” to version “y.” With the addition of ownership, we could mark this date as running an “event” of an Oracle database platform upgrade. This new event would be “owned” by the database update team.

    The block of time the team outlines for the Oracle upgrade “event” would then take on two additional characteristics for teams in general: inclusionary and exclusionary implications. As an example, the time set aside for the tasks to upgrade the Oracle platform becomes inclusionary for the owning team. Members of this team would be allowed to schedule tasks during this block of time, ideally intended to facilitate this goal of upgrading the platform.

    Other teams are (by default) excluded from performing actions during this block of time so they don’t impact what the database team is attempting to accomplish. When other teams pull up their respective calendars to schedule some new task or event, the April 27 block of time is greyed out or presented as unavailable because the database team owns an event listed in that block of time. For the database team the time is inclusionary, for all the other teams the block of time is considered exclusionary.

    The benefits to you the customer are immediately visible. By adding ownership to the calendar construct, events could be scheduled in such a way that teams do not inadvertently “step on” each other (i.e., attempt to schedule tasks during the same window of time that would adversely affect the other). It also allows you the customer, to view a dashboard that shows the “kinds” of events that are scheduled throughout the week, month or quarter, and who is doing what.

    To avoid being too strict with this idea, the RO tooling vendor need only add concept of granting exceptions to requesting teams by the owning team for events that would not impact their efforts. So as an example, if a different team wanted to work on tooling the SQL Server platform during the same block of time that Oracle platform was being updated, and the servers were different anyway, there is no reason why that team could not schedule a similar event of its own at the same time. If ownership is the only concept added in the RO tool, an exception would have to be granted by the Oracle team, but this would be costly in terms of administration. It is for this reason we introduce the second main construct in the intelligence of calendar management.

    The Intelligence of Component Awareness

    The second main concept of intelligent calendar management is component level awareness. This dimension of awareness is based on identifying specific “technologies” as resources or components that are part of the environment you operate at your company. It doesn’t have to be a ground-up exercise, and if you prefer one of those, you could use a standard discovery tool to keep it up to date. But the more basic idea is to simply identify the fundamental technology components you use in your world.

    So as an example, I could identify that this company uses Oracle databases (perhaps also tracking the version if there are multiples of it), SQL Server databases, the type of servers it uses, the operating systems, the middleware, perhaps a mainframe platform and the network fundamental technologies, etc. It is highly likely that there are groups of people who are uniquely identified in this company that manage these specific types of technologies. This makes the associations easy for event ownership. The Oracle guys become associated with the Oracle technology components but not with the SQL databases, etc.

    Having teams associated with the components they manage allows for better parallel event scheduling. But this is only the first level of intelligence. The next level is to track, by application, the underlying technology components they utilize. This process is done during the build and deploy automation anyway, so the effort is nothing more than what is done today. The beauty of having this spider web map, or this map of technology component dependencies, is that it adds a new level of capability to scheduling in our calendar.

    At this point, scheduling an Oracle event does not preclude the SQL team from scheduling another one during the same time window, since those two technology types don’t conflict with each other. Both teams would see open calendars on the same day due to this intelligent calendar awareness of component interdependency, or lack thereof. This dimension of intelligence precludes having to grant needless exceptions to owned events. The only requests for exceptions would come from users or teams that actually would impact the upgrade event, and they could be evaluated for legitimate need and potential real impact.

    Let’s return to our Oracle platform upgrade event our mythical company scheduled for April 27. With the addition of component-level awareness, when this event is scheduled by the Oracle database team, every application that uses Oracle is “automatically” notified that this time window is now off-limits for its development or usage activities. This limitation could include automatic notification to application development teams that they could not deploy new code during this window. Or the notification could be expanded to notify users of the impacted application(s) that this list would be down during this window of time. Applications that do not use Oracle would remain unaffected and not notified.

    Imagine what this kind of automatic notification could do for both development and the user base of the technologies you operate, all coordinated automatically through the RO tooling you use today in DevOps. Then think about the possibilities from here. The Oracle team could look at a block of time again and examine whether scheduling on April 27 would have a high impact on other teams, or not, before it finalizes it. If the impact to others is high, perhaps the team picks a different date. The possibilities become endless.

    The Intelligence of Scope

    The third and final architectural dimension to add to calendar intelligence is the concept of scope. Scope is a simple hierarchy that you could define about your company by logical business subdefinition. As an example, you could outline the scope you define as: global (entire company), business unit (listed by name), product unit (listed by name) or departments (listed by name). Just using these kinds of simple scope definitions begins to insert an impact assessment into the event scheduling that takes place.

    As an example, events could be scheduled with a global scope, owned by the change management governance body, that excludes a 10-day window because of a corporate audit by outside regulators. Now, when any development team attempts to schedule deployments or releases, for example, their calendars are automatically greyed out during this 10-day window because of the corporate governance event. Exceptions would only be granted by that governance body. In this case, this prevents accidental changes occurring during the middle of an audit.

    If the example cited above is too broad, and the mythical company only needs to limit an event to a particular product line or further down to a particular department, the “scope” characteristic allows them to do so. This final dimension to intelligent calendar management does require a higher level of sophistication from the RO tooling vendor and a higher upfront investment in the administrative costs of providing DevOps services. But for an enterprise-class installation, this is an investment that can take the chaos of corporate communication where it comes to the impact of change to a systemic nirvana of communication on demand.

    The Impact to Your Bottom Line

    So why outline an architecture that most RO vendors have yet to implement? If it is candy I created that you can’t have yet, why discuss it? Two reasons: First, when you begin to examine the cost of running your DevOps set of services, you will find that, like many other enterprise-class efforts, it is the administration costs that are heavy at the outset and increase over time. The effort it takes to manage “communication” and avoid having interdependent technologies “step on” each other is no easy task, particularly when we have moved from weekly deployment windows to potentially daily deploys to production across large app portfolios or complex legacy hardware architectures.

    Second, for those of you who still maintain data centers, consider how much time you “waste” if your Ops teams continue to require a 12-hour to 48-hour window over the weekends to make updates to their own platforms. Going to an external cloud provider is a way out of this, but for organizations that are unable to fully do so, the weekend downtime remains a significant issue. DevOps will create desire in the Dev teams to want to use this traditionally Ops window for their own purposes as well. Once DevOps has been fully embraced, there is as much a need to automate communications and intelligently ensure we do not step on each other as there ever was. DevOps accelerates this need, as it accelerates the speed of innovation itself.

    The proper tooling, or rather the tooling that is thinking about intelligent architectures that will greatly reduce your costs and your overhead and ensure consistency in your world—these are the features (and the thinking) you need your tooling vendor to embrace. The alternative is having to pay your labor base to manage it another way. If your leadership and your vendor partners are not considering these issues, they should be. If you need help, feel free to contact me (I am still looking at the moment). The goal of DevOps to automate process should not just be about moving code; it should extend to how we administer, communicate and inject governance as well. Intelligent architectures accomplish these goals, and any vendor could be there first, if they wanted to.

    To continue the conversation, feel free to contact me.

  • The Beauty of Inheritance Architectures

    The Beauty of Inheritance Architectures

    Spoiler alert: This article is not about inheriting your great-grandparent’s estate. It is rather about how embedding the principles of inheritance into the architecture of your chosen DevOps toolset can yield exponential results. And yes, I realize that you, the customer, have little to do with what a vendor builds its tooling architecture upon. However, you, the customer, can vote with your feet, to search out products that offer features such as inheritance, rewarding vendors that have it and punishing those that do not. The simple question to address here is, Why should the customer care?

    Like anything that can grow, sustainability on scale is a key concern. How much will it cost me when I am maintaining 6,000 applications on this particular DevOps toolset, as opposed to the six applications I started with? You can also infer maintaining a higher number of technologies as having the same need. For example, if I need to maintain three different types of databases, two different middleware, four different hardware platforms on “x” different versions of operating systems … you get the idea. The larger the application portfolio (in sheer numbers of apps), or the more technically complex the portfolio (in sheer numbers of technology components in an app), eventually will drive you to worry about how easy it is to maintain these applications within your chosen DevOps toolset.

    The capabilities of your chosen set of DevOps tooling will govern the costs of both the initial adoption of your services as well as the ongoing maintenance of the application portfolios you serve. It is hard to sell the value of DevOps to the business when it begins to see a one-for-one increase in DevOps engineering staff for every new application or technology venture under consideration. However, one of the chief culprits in driving large DevOps support staff organizations boils down to the lack of an inheritance architecture in the tooling that was originally selected.

    Setting a Common Language

    So before we discuss how inheritance will benefit us, first let’s agree on a common vernacular of what an “application” is. When you say the word “app” to someone, you get a wide variety of responses on what that means—from something that sits on your phone to a cluster of things that sit on the mainframe. Let’s begin by examining what an “app” is by the effort it takes to onboard and then maintain it on our DevOps sets of services (regardless of platform).

    Consider for a moment the hierarchies by which we could describe software constructs. It is relatively easy to discuss how software works in terms of hierarchy. First, at the lowest layer, we could describe having multiple technology components (or the lowest level of a particular piece of technology that must be addressed in a build). Next, assembling these components together forms the picture of what a single application looks like. Finally, assembling multiple applications that have a common symmetry or common purpose together form a system or series of apps.

    An imperfect but quick example could be the Microsoft Org Chart component of its Office application(s). For purposes of example we will argue that Org Chart, Clip Art Manager and Equation Editor are all components of the application Microsoft Excel. To build a new version of the application Excel, we would have to deal with each of these individual components that, once assembled in our build, would create a new version of Excel. When you combine more of the apps—Excel, Word, Outlook and PowerPoint—you get the system, or series, called MS Office. (And, yes, I know this is a less than ideal example, but it is meant only to illustrate a point.)

    Assessing the Work Effort

    The reason for discussing software in this way allows us to begin to discuss the level of effort to automate it. By addressing the work effort to automate each component (by type), we can begin to estimate how long it will take to complete “the automation of software.” For example, if my homegrown application has a source code component, a configuration file component, a database component and a middleware component, then I have four components I must automate (or include in the build and deploy process) to assemble a new version of my application.

    This gives us a “least common denominator” language for assessing how much effort it takes to automate a given app. I have seen most common applications have only two to three components. But I have also seen several apps that, due to their unique architecture and construction, have 30 to 50 components a piece. Obviously, it is easier to automate an application with only three components than one with 50 components. The other factor we need to apply on the ease of automation is whether I have encountered this particular component technology before in my prior automation efforts.

    Let’s say, for example, that the application I am estimating contains Java component(s). If I have encountered Java automation prior to this, I should have constructed a template for Java type component automation that I can use for a situation just like this one. In this situation a familiar technology component is far easier to automate than a new one (or, rather, new to me). However, if this is the first time I have ever seen Java, then it may take me a bit longer to construct the template I want for this app, and for future apps I encounter, that also use this technology type.

    The Beauty of Inheritance

    Here is where inheritance begins to play its critical role. If my DevOps-build construction tooling includes inheritance, then I could construct a “master” template or framework for each type of technology as I encounter it. A master template for SQL databases or Oracle or DB2, as an example. Or, I could make a master template for WebSphere or SharePoint types of technologies—one master for each kind of programming language we have in my shop. You get the idea.

    Having done this allows me to have a limited number of master templates, in which I can maintain all the critical steps and processes my company wants me to do for a given technology type. If different business organizations or development organizations need a variation, they can create a “child” of my master and vary it according to their own needs. As long as the linkage is maintained to my parent/master template, I can make the big, overarching changes driven by corporate mandate, and the business unit can continue to vary items to their unique needs.

    Having inheritance in this instance allows me to roll out changes to an entire organization, such as, for example, a security patch or process, for a given technology by simply adjusting the master template of that component. Inheritance automatically flows my change down to all the “children” of my master and possibly to all their “children,” should the need arise. The alternative is having to go through the specific automation of each application that has the targeted technology component and updating it manually, hoping I don’t miss one, and hoping the steps I may need to adjust do not break the old automation instead of alter it the way I needed.

    Imagine having a WebSphere subject matter expert able to adjust every client application that uses your corporate installation of WebSphere by simply adjusting one master template. Or having a network guru, or database expert, be able to do the same for all applications that use their particular technology component all at once. Inheritance preserves the real-world need of having limited alterations of the same technology type within the same company, flowing down changes from parent to child.

    Having a one-size-fits-all is simply not practical. Using copy/paste requires repeating updates to each impacted template or framework manually. But having the ability to create offshoots from a main, and not lose the connection to the main, allows corporatewide alteration from a central point of control. The costs of maintenance or administration are formidable; inheritance can reduce them. This allows a much higher ratio of applications or technology components supported by a much lower number of engineering staff.

    A Wider Impact for Inheritance

    Inheritance is keenly useful in release orchestration software as well. Most vendors offer the cut-and-paste approach: you copy the last the release you did, alter it slightly and use it again. That might work for some organizations. But having an inheritance feature creates entirely new dynamics on how a release is constructed for a given line of business and how corporate governance can be implemented with minimal effort. In this case, a corporate master release template can be constructed, and child releases can inherit from the master all the main governance required without risk of omission.

    I realize inheritance architectures might not be at the top of your list when evaluating DevOps tooling solutions. But I would submit it is a strategist’s job (if you need one, let me know, I am looking 🙂 ) to point out how long-term sustainability can be greatly benefited by this architectural feature—even if the vendor would love nothing more than to steer you away from it. If you already have committed to a tooling solution, you may want to bring this up at your next vendor features performance meeting. If not, you should at least consider what having it means to your internal cost structure, and what not having it will add to the true cost of ownership of any proposed solution.

    To continue the conversation, feel free to contact me.

  • 5 Prerequisites for a Successful DevOps Initiative

    5 Prerequisites for a Successful DevOps Initiative

    There are certain tools and attitudes that are essential to any successful DevOps transformation.

    The manner in which a DevOps transformation is performed will have huge implications on the level of agility the company is able to achieve. However, before that point is even reached, some vital foundations must be laid. It is from that foundation that any innovation occurs. With that in mind, each of these elements are equally important, and should be regarded as such.

    Consensus

    Your first achievement on the road to successful DevOps should be an agreement on what DevOps means to your company and how it should be applied to your needs. Consensus on a strategy is almost as important as the strategy itself and must be in place before a detailed road map is constructed. Remember that DevOps is subjective and its implementation should vary from company to company. Consensus hopefully will be easy, but if not, it must be built by persuading doubters of the need for certain principles and practices selected.

    This is the first step on a cultural journey; the collaboration between Dev and Ops staff and their consensus on a plan could make or break the DevOps idea. The concepts behind the initiative need to be agreed on from all corners of IT, as well as the business and any other key stakeholders such as suppliers. It is important the cons and pros of switching to DevOps are made clear for a balanced proposition. The argument for a shakeup on this scale needs to be compelling and justified for it to go ahead.

    Consensus also involves handling expectations. To keep everyone happy there must be metrics and approximate timelines set in place so everyone knows what to expect and when, to avoid conflict further down the line. Only once consensus has been achieved and the basics are put in place can a more detailed road map be built.

    Flexibility

    A thorough road map is the skeleton that allows its muscular teams to flex independently toward a common goal. However, flexibility without allegiance to the agreed principles can cause chaos. The DNA that runs through all teams stays the same but applications vary, and so teams need to work autonomously if they are to cater for the specific needs and nature of the app they are working on.

    While the road map gives a structure and handles expectations, DevOps culture needs to be tied to business outcomes and not formal internal milestones. Each DevOps team needs to be trusted to interpret the road map as it suits the specific project they are working on. Agile methodologies must be used, of course, but their interpretation is devolved to the teams themselves. This license and trust is key in allowing DevOps teams to react quickly and mold around the needs of the individual application.

    Teams need freedom in terms of what tools they use and to an extent, how they use them. The methodology used to achieve the agreed core principles should have an element of flexibility. Tool-agnosticism is a must when choosing an automation solution for effective DevOps. Teams are happier and work faster when allowed to choose the tools that suits them best, making them more successful.

    Automation

    As C. Little said, “DevOps isn’t about automation, just as astronomy isn’t about telescopes.” An automated continuous delivery pipeline should underpin any DevOps initiative, adding consistency and reliability to flexible delivery methodologies. Tribal knowledge from Dev and Ops is gathered to create a pipeline that is the sum of the knowledge of its contributors. The pipeline is repeatable, and over time is made more efficient by rooting out inefficient manual processes. Modelling into a workflow helps all involved visualize where a team is in the app life cycle and what next steps need to be taken next and by who.

    Trust in application release automation can be a big deciding factor in how companies achieve successful DevOps practice. Sometimes there is a culture within a company of wanting to have manual control over certain processes that inevitably slow things down. Automating any and every manual task possible is essential to speed up DevOps. If a manual intervention is needed, the relevant member of staff can be notified by the automation solution itself, which automates out the remainder of the task for which manual intervention is not required. Over time, manual interventions will disappear altogether.

    The ability to give DevOps teams freedom in their tool selection keeps staff productive but adds yet more complexity and leads to a highly heterogeneous workplace. It’s important your automation solution has approved integrations to all tools DevOps teams might choose and also releases free integration to new tools quickly, so you can stay on top of the latest trends and continue to be successful.

    Cooperation

    Unless you’re a startup with heavy investment, you’ll likely be changing the structure and workload of your existing staff rather than hiring a new team suited perfectly to their individual roles. The success of DevOps depends on the ability to make positive change in the way team members collaborate.

    Resistance to change may occur due to staff being accustomed to working in one particular way for a long time. However, this could also be down to personality type. As it is Ops and Infrastructure staff who face the biggest changes and are used to limiting rather than encouraging risk, often they can be the most resistant to change. Often devlopers do not have to change the way they work to the same extent, so fewer feathers are ruffled.

    I have seen big corporations that do not have compliance from everyone, so their DevOps is limited in scope and unfortunately they will never reach full agility. Sometimes you must adjust your teams to ensure positivity to DevOps and receptivity change in general.

    Rethink Architecture

    I’m not saying DevOps cannot be implemented in legacy systems, but there are elements of legacy systems that make it hard to instill full automation and continuous delivery. What is important is that the organization is open to a certain degree of architectural change. Monolithic apps can certainly slow down a DevOps initiative—often it’s mainframe apps that are not built with a service-orientated architecture—so it’s important to know how to adapt them to fit your strategy.

    A full architectural revolution is not needed to be successful. Obviously this is where cloud-native startups have an advantage as they can assemble their infrastructure from scratch, but there is no need to rip and replace existing systems for the sake of it and waste money. Certain elements of even the most monolithic systems can be modernized—for example, Siebel deployments can be fully automated so operations staff are free to concentrate on more demanding aspects of the cultural change DevOps involves.

    DevOps applications can interact with legacy systems if we use the right techniques. Make sure your automation solution has integrations with the likes of SAP, Oracle and Siebel, and so enables legacy systems to be enablers on your digital journey rather than obstacles that will stand in your way as you seek to reach full DevOps and end-to-end continuous delivery.

  • Humans in the Machine of Release Orchestration

    Humans in the Machine of Release Orchestration

    Let’s face it: Most organizations are still struggling to achieve the Nirvana of full gated automation. At first, we automate the core of our DevOps services, the build and the deployment disciplines. Over time, we realize that a templated or framework (or pick your similar terminology) approach toward automation allows for scale as well as predictability, lowering costs. It is here where the lion’s share of variability can be reduced and scalability achieved.

    The next culprit to hit the radar is usually test automation, but this tends to be where the rubber meets the road for the level of work it takes to automate process. So generally minimal work gets done, ascribing to the 80:20 rule that 80 percent is quickly attainable and should be sufficient, the remaining 20 percent is just too hard. But having gotten this deep in the DevOps maturity scale, release orchestration (RO) is usually next up in our continuum of services.

    The automation of release orchestration is a different animal than the core of build, deployment or even test automation. Release orchestration is more about automating our process of change management. Invariably the spreadsheets or SharePoints of yesteryear are intended to give way to our preferred vendor’s methodology of reinventing the release wheel in their proprietary toolset (albeit with better results expected). But usually this is the wrong approach as well.

    Changing Human Thinking About RO Construction

    As I continually repeat, DevOps is about thinking differently about “how we innovate.” (If you need more strategic leadership help in this regard let me know, I am looking.) Consider that our spreadsheets and SharePoints, while impressive, constricted us to thinking about release and change management within the architectural constructs or limitations of the spreadsheet and SharePoint tools. They were task-oriented, less interactive, top-to-bottom lists of what we thought we needed to do in any given release. If we were to truly go back to the drawing board and thought about creating a release without those constraints, or just with the idea of parallel tracking, what might we accomplish?

    Digesting the release process with a lean mentality breaks down large sections into more manageable subsections. This reduces risk, and increases speed, particularly if it is coupled with parallel execution. If there truly is a sequential dependency in our tasks, let’s call it out, but when we do, we must ask ourselves whether that dependency is a go/no-go for all activities, or for only some of them. If it is the former, so be it; if it is the latter, then perhaps we can adjust our sections to account for this and run more of them in parallel. Just implementing a parallel mindset can radically reduce the time it takes to run a release from end to end. But we must also ask ourselves, What other key features of any preferred release orchestration toolset can alter how we think?

    Changing Human Thinking About RO Execution

    The other axis upon which to digest release orchestration is in its real-time interactivity. Consider for a moment that some tasks either cannot or will not ever be fully automated. Some tasks will remain in the province of a human to fulfill. For example, (and I cannot imagine why, but humor me), if we needed to install additional hard drives in bare metal servers during a release, the physical installation would always require a human to do so. With old release orchestration technology we must wait for shared spreadsheet files to be updated by the engineer when he/she completes this task (or any other manual task example you can think of).

    With new mobile release orchestration technologies, the engineer could mark them complete (or percentage complete) in real time on his/her mobile device while still in the data center, standing in front of the hard drives/server. How much time does this save, in pure walking to his terminal, logging in and updating a file, emailing it to the change manager, merging it with the current one, etc.. A simple swipe and done. The engineer is now able to provide a great deal more information to the team in real time, and with minimal effort. This gives real-time visibility to the release manager such as has never existed before.

    It can also grant them a level of direct interactivity. Imagine the hard drives’ physical installation has been completed, but manual testing is failing. If the engineer has no good way to share the information with the team, risk for overall on-time completion cannot be properly assessed. Proper tooling avoids the scenario where change and release managers are waiting on a task, and drill-down activity is unavailable (picture a spinning icon, rather than a percentage complete on specific task item). Add the dimension of red-yellow-green to track the completion of tasks, whether parallel or not, and we have a dashboard that calls our attention to the problems impacting a release in real time. This is far less possible with older RO technologies until somebody checked their watch and started making phone calls.

    Changing Human Thinking About RO Valuation

    Now every organization dreams of a release containing nothing but virtualized full-stack deployment for every app (even the legacy ones), where everything from bottom to top is automated during the deployment process. However, some of us living in the real world, still have a few database scripts that must be run manually during a production deploy (don’t ask me why). Or pick your poison in the real world you are contending with, often infrastructure and platform tasks are the last to be fully automated.

    So during a release, engineers need to be on hand and ready to go when called upon. If however, we now all subscribe to a tooling solution that acts as a common backbone for our release orchestration … the engineers know exactly when to do something, if they can do it early, and when they can go home afterwards. There is the value proposition for the engineer, less time spent doing tedious work during a weekend, when they would otherwise prefer to be at home.

    The value proposition to the release manager is real-time insights into all this activity during a release. The automated tasks flow through just as you would expect them to, with up-to-the-minute status of their progress until completion is reached. The remaining manual tasks provide as-good-as-it-gets status updates, also in real time, and serve as the next target of opportunity for automation prior to future release(s).

    Changing Human Thinking About RO Perspective

    One thing that always seems to be missing from a number of vendor’s offerings I have reviewed in the release orchestration marketplace, is a view from human centricity. Most release orchestration tooling is obsessed with task management, integrating it with calendar scheduling and designing it for maximum efficiency in getting the next step done. If however, our vendor community would embrace the idea of intelligent calendaring a bit more (a future article will discuss this characteristic more in detail), the offerings could form a more human-centric view of the release orchestration process.

    Imagine what it would be like for either an engineer or a change manager to log in to the release orchestration toolset and have a personalized dashboard outlining the tasks that need to be approved, worked on or completed to support the all pending releases for that user (based on integrated workflow and intelligent scheduling). An automated red-yellow-green attention draw, for work that may be behind, on schedule or completed early, would greatly improve the human experience of release orchestration. And an integrated departmental or team-focused view of the work effort from manager or employee-centric views would add to job performance critiques—proof of superior, or inferior performances over time.

    This represents only a simple recognition that humans should not be stuck in the machine of release orchestration tooling, slave-to-a-machine-centric view of actions to be completed. Instead, a vendor could recalibrate a more human perspective—a subtle alteration in perspective adds an exponential lift to the value of this release orchestration toolset, and it seems to me, any vendor could be there first … if they wanted to.

    To continue the conversation, feel free to contact me …

  • DevOps killing outsourcing? Another point of view

    DevOps killing outsourcing? Another point of view

    After reviewing the articles (here and here) on DevOps driving the end to outsourcing I felt it would be constructive to provide an alternative point of view.  The premise of the article is that because DevOps requires close collaboration between development and operations that it would be difficult if not impossible to effectively implement DevOps in a heavily outsourced environment.  Additionally the productivity advantages of DevOps would help eliminate the need for outsourcing.  Therefore, organizations wishing to see the advantages of DevOps should move away from outsourcing.  While I can see the advantages of that approach, I also think that companies that have to live with outsourcing can’t ignore the benefits from implementing DevOps principles with a well-designed deployment pipeline.

    From my perspective, companies looking to implement DevOps first need to start by understanding their architecture because it has such a large influence on how to effectively implement DevOps.  If they have loosely coupled architectures that enable small teams to independently develop, qualify, and release code then they should look to the world of unicorns for how to implement DevOps.  They should look at integrating operational people onto the development team and work to evolve the culture by blurring the lines and responsibilities between the two groups.  This feels like a fairly straightforward process that many unicorns have proved to be very successful.  For these types of organizations, I tend to agree with the articles recommendations that DevOps would work better without outsourcing.

    The problem is that most large traditional organizations looking to implement DevOps don’t have the benefit of an architecture that enables small nimble teams to work independently.  They typically have very tightly coupled legacy architectures that require 100-1000s of engineers to work closely together to develop, qualify, and deploy complex integrated systems.  Ideally, they would re-architect their applications so they could see the benefits of nimble independent teams.  The reality though is that this is just not practical for most traditional organizations in the short-term.  Additionally, they can’t wait for the re-architecture before they start realizing the advantages that DevOps principles can provide with a well-designed deployment pipeline.

    For these organizations implementing DevOps principles (the ability to release code to the customer on a more frequent basis while maintaining or improving stability and quality) is more about creating a well-designed deployment pipeline that builds up a more stable enterprise systems on a regular basis so it is much easier to release the code on a more frequent basis.  This is done by creating a deployment pipeline that integrates the code across the enterprise system on a much more frequent basis with automated testing to ensure that new functionality is not breaking existing code and the code quality is kept much closer to release quality.

    From my perspective this approach to DevOps is rather different from the more unicorn type approach described in this article.  It though does address the biggest opportunity for improvement that does exist in more large traditional organizations which is coordinating the work across teams.  In these cases the working code in the deployment pipeline is the forcing function used to coordinate work and ensure alignment across the organization.  If the code from different teams won’t work together or it won’t work in production, the organization is forced to fix those issues immediately before too much code is written that will not work together in a production environment.  Addressing these issues early and often in a deployment pipeline is one of the most important things large traditional organizations can and should be doing to improve the effectiveness of their development and deployment processes.

    These benefits are even more pronounced when working in organizations that have outsource development across multiple companies around the world.  Ensuring the different vendors don’t go off track and create code that won’t work together in production is critical and a well-designed deployment pipeline is a critical tool for coordinating development across different outsourced organizations in traditional tightly coupled systems.  For these types of outsourced organization, DevOps principles while different are almost more important than they are for companies that do all their development internally.

    About the Author/ Gary Gruver

    gary gruverGary Gruver is an executive with a proven track record of leading Large-Scale Agile transformations and co-author of “A Practical Approach to Large-Scale Agile Development” and the new book, “Leading the Transformation: Applying Agile and DevOps Principles at Scale.” He is currently working to help organizations with their transformations after successfully leading two large transitions at HP and Macy’s.  As the Director of Engineering at HP, he led a large organization (400+ developers) on a journey from waterfall to agile development. At Macy’s.com, as the VP of QE, Release, and Operations, he led the transition to Continuous Delivery.  Reach him on twitter at: @GRUVERGary