devopsy.kinsta.cloud

Tag: Enterprise Adoption

  • Don’t Let Your DevOps Transformation Be a Parade of Tools

    Don’t Let Your DevOps Transformation Be a Parade of Tools

    Choosing the right DevOps tools should be a careful, thoughtful process.

    On May 12, 1848, a store owner named Sam Brannan held a “one-man parade” to announce the start of the San Francisco Gold Rush. “Gold! Gold from the American River!” Brannan shouted up and down the market street in San Francisco. He held his hat in one hand and waved a bottle of gold dust in the other. San Franciscans had received false news of gold before. But by all accounts, Brannan’s performance sent residents running in search of riches. 

    Brannan had a good reason for spreading the news rather than panning for gold himself. The canny entrepreneur owned a general store that served the workers, and in the week between learning about the discovery and yelling about it in San Francisco, he’d bought all the picks and shovels in the city. Brannan’s announcement helped spur a seminal event in California’s history. As Brannan raked in money selling mining supplies, his actions also, years later, led to the coining of a famous maxim. (source)

    During a Gold Rush, Sell Shovels

    In the current age of DevOps and Agile rush of enterprises, there is a lesson to be learned from the gold rush: Don’t let your DevOps transformation become a parade of tools. While automation is critical for the success of modern ways of technology delivery and operations, the wide range of tooling choices can often lead to wrong decisions—especially well-intentioned, top down purchase of tools, duplication in toolsets and, most importantly, ignoring the voice of the teams on the ground.

    With such variety of tools available in the market today, we all have opinions regarding the tools that we would like to use. Our preferences are based on either our history of using certain tools, because we have invested time and effort to learn a tool, or partnerships your service vendors might have with tooling vendors.

    This leads to two general attitudes toward tooling adoption, with each having their own advantages and disadvantages:

    Autonomy for teams: Let teams decide the tools they feel are suitable for their context.

    The advantage is that teams can experiment and pick a tool that suits their context better. The disadvantages can, potentially, include security risks due to lack of patching and maintenance, and limited cross-team collaboration and sharing of tooling skills and know-how.

    Standardized tool suite: A centralized way of dictating tooling choices.

    The advantage is that it allows for capacity planning, integration between tools, larger pool of community for collaboration and create platforms-based solutions.The disadvantage is that this method at times prevents teams from using tools that best suit their unique needs and creation of avoidable dependencies on centralized tooling teams in the organization.

    These contrasting views lead to debates at every level for the choice of tools and, in some cases, slowing down decision-making—or even leading to the purchase of tooling licenses without input from technical teams on the ground. The decision-making process must be improved, and the need for autonomy must be balanced with standardization and optimization. Also, the two approaches of “autonomy for teams” and “standardized tool suite” do not always have to be binary and will change depending on the context of the team needs and area.

    By considering desirability, viability and feasibility of the DevOps tools, we can help achieve the balance between autonomy and alignment.

    Here are some tips to help ask the right questions:

    Desirability

    • Do teams want this? or is this a solution chasing a problem?
    • Do you anticipate that teams will need this in the near future?
    • Does this help improve team productivity and effectiveness?

     Viability

    • Does this make commercial sense?
    • Have we considered maintenance costs and total cost of ownership?
    • Do we have existing license arrangements with the tooling vendors?
    • Do we have comparable tools already in use and that can do the job?
    • Do we need to purchase licenses throughout the enterprise?
    • Are there any open source alternatives?

    Feasibility

    • Can we implement this in the enterprise IT landscape?
    • Have we considered the ease of setup and configuration of the tool?
    • Does it scale across the enterprise? Does it have to scale across the enterprise?

    — Aditya Vadaganadam

  • Has DevOps Caused the Re-emergence of Shadow IT?

    Has DevOps Caused the Re-emergence of Shadow IT?

    Shadow IT is back, and this time, it’s in the DevOps toolstack

    “One of our clients conducted an audit and found more than 10,000 Jenkins instances across their IT landscape!” This remark from a tooling vendor that helps enterprises centralize their continuous integration infrastructure really took me by surprise. For starters, I was impressed how the company had unleashed creativity at this scale—if the number of Jenkins instances in an organization is an indicator of DevOps. Clearly, its employees have taken the saying, “Don’t ask for permission, ask for forgiveness,” very seriously.

    However, there are certain risks with this approach, especially security risks if teams haven’t considered the long-term implication of their decisions. For instance, how do you handle another major security flaw such as the OpenSSL Heartbleed bug? You may have to send hunting squads to track down all the instances of Jenkins (in this case) across your IT estate—including the Linux server hidden away under the desk of the engineer—and patch them. If you are lucky, you can audit all the virtual machines on your chosen cloud where all your tooling instances reside. And if you are really lucky, the entire DevOps tooling infrastructure would have been set up in a DevOps way, using infrastructure-as-code principles with automated patching capability.

    Consider the nightmare scenario for IT compliance, security and governance teams: The spectre of shadow IT is amplified due to poor adoption of DevOps! Shadow IT in the application stack has always been a big headache for CIOs; the use of IT systems within an organization without the approval, or even the knowledge, of corporate IT has never gone away. Shadow IT has been historically associated with business users who made the purchase, but did not have the expertise to maintain and upgrade the software to make sure there were no risks to the wider organization. Much has been written about the problem in the application stack and how DevOps ways can help resolve the issue.

    However, with the proliferation of DevOps tools in a localized way, the shadow IT problem is now manifesting itself in the tooling architecture of enterprises. With the increasing adoption of DevOps practices and tools in the enterprise, teams have started installing various DevOps tools in a localized and autonomous way. This is leading to the shadow IT problem in the tooling landscape of enterprises. It’s slightly different than the problem that originated in the application stack, but with similar risks and challenges. Access to cloud in the enterprise has made it easier for teams to set up their own localized DevOps tool sets. A sizable program of work in its full DevOps glory can easily use dozens of tools and need to consider aspects of security such as:

    1. What is the patching policy ?
    2. Are your DevOps tools critical to the business due to the dependency on them for the build and deployment of critical  services that the business requires?
    3. How resilient is the tooling infrastructure ?
    4. Is there enough knowledge in the enterprise to handle this change?

    Essentially, there are four risks of shadow IT due to DevOps tooling, based on a pertinent classification by Computer Weekly:

    1. Software asset management: The risk of discovering unapproved and unlicensed software is huge and the CIO could be potentially jailed for sanctioning usage of unlicensed software.
    2. Compliance with industry and government standards to demonstrate quality to customers.
    3. Lack of testing and change control: Managing cycle of change and releases becomes complicated when there are dependencies on third-party SaaS.
    4. Configuration Management: IT teams rely on the veracity of information in the configuration management database CMDB and shadow IT compromises it.

    Organizations typically respond to the problem of shadow IT with rationalization, migration or even sunset of the targeted applications—most of which are reactive measures. In the context of DevOps tools, an extreme way to deal with the problem is by taking control of all tools with an “iron fist” and shutting down everything—at the risk of killing off innovation in the enterprise. In the short run, teams do achieve higher levels of productivity by deploying tools in the way that suits their business context and experience of the engineers in the team. However, over time I find that the teams are overburdened by the maintenance overhead of DevOps tooling and maintaining a tooling stack in addition to their software application stack. At times, if they are early adopters in their organization, they tend to be victims of their own success: by supporting other teams with their tooling solutions, almost becoming a tooling platform provider internally, and diverting their attention away from their core purpose of delivering value to their business partners.

    One pragmatic way to resolve this is to add an element of centralization or “platformization” of the tooling infrastructure.

    Autonomous teams understandably fiercely resist any attempt to centralize services in the enterprise. A few useful ways to gain a balance between innovation and control include:

    1. Provide a responsive tooling platform: “You can have any color as long as it’s black” approach to DevOps tooling will only lead teams to finding their own unique solutions.
    2. Anticipate tooling needs and build them: Think ahead, track the trends and procure the tools that you expect teams to achieve. For example, if a few teams are already playing with docker and containers, then think a step ahead and look at Kubernetes to manage the containers.
    3. Focus on user experience: Think about the ease of subscribing to tools, installation, billing … the entire life cycle
    4. Offer help to transition to the tooling platform.
    5. Run an internal communications campaign on the tooling capabilities that are available to teams.

    Ultimately, a well-thought-out DevOps approach and implementation can actually help resolve the challenge of shadow IT that organizations face, instead of accentuating the problem.

    — Aditya Vadaganadam

  • Changing Compensation Before it Kills Success

    Changing Compensation Before it Kills Success

    Have you ever seen the sign, “Will work for food”? If you deconstruct the premise a bit, we all work for the rewards systems that interests us. We “trade” our labor (ideally, focus our passion), for a set of acknowledgements that range from survival to pride. And ultimately it is our employer (whether the boss or the board of directors) who decides what our rewards systems will be.   That is the common sense part. But when something such as DevOps enters the picture, it alters traditional thinking throughout an organization, and the ripple effects should (and eventually will) reach the HR department.

    The most immediate effect of DevOps on HR occurs around talent acquisition. Finding a qualified DevOps engineer that it is possible to afford—and the company can retain—becomes the unicorn of recruiting departments from one coast to the other. The alternative is creating or adopting a training curriculum that can move a traditional programmer or engineer into a fully qualified DevOps engineer. Of course, advice varies on exactly what that is, and if it’s successful, your organization becomes a feeding ground for every other competitor to poach your folks right after training.

    Because this HR acquisition problem is so great for most organizations, they can overlook some other underlying issues that continue to occur. These problems have always been there, but DevOps brings a spotlight to them that did not exist prior to its implementation. As stated earlier, we “do” what we are “paid” to do. In organizations that innovate in the software discipline, we have always “paid” that entire division to innovate. Prior to DevOps, the pace of that innovation, was slow and manageable. Once DevOps begins to take hold, the pace of innovation begins to enter the 400mph superhighway, where traditional control mechanisms no longer work.

    In itself, the success of the controls over change—transitioning to a high rate of speed—will be determined by how we implement DevOps. It is the other half of DevOps that needs some focus. While our Dev organization is cranking out change exactly as we have always paid them to do, our Ops organization traditionally has been paid for exactly the opposite characteristic. We pay our Ops organization for stability. We want the trains to run, on time. We do not want a 2 a.m. phone call from an angry CEO—or, much worse, an angry customer—because our operations have “broken.” And, of course, the No. 1 culprit for causing breakage in operations is … [drum roll, please] … “change.”

    In the yesteryear of Waterfall construction and manual delivery, our Ops teams had time to digest what change was coming, prepare for it, adjust to it and get it implemented smoothly (or at least pretend to). In the current day of Agile construction and DevOps delivery, I might have moved six new builds into production in the time it has taken you to read this article. Ops has no time anymore to digest and prepare for change, so if the quality of that change is not significantly better than it used to be, the angry-phone-call potential goes up exponentially. And yes, there are technical remedies to ensure quality (I will reserve discussing those for another time).

    But the tradition of how we compensate remains in place. We comp Dev to innovate, and we comp Ops to remain stable. This makes not only silo thinking, but also mortal enemies during the adoption of DevOps into an organization. All of a sudden, each views the other as potentially causing harm to their ability to continue earning compensation. And they are right to see it that way, as HR has yet to alter the models to create partnerships instead of combatants.

    To fix this, the compensation models must be adjusted to pay Ops teams to get involved in the software development lifecyle (SDLC) process from the upfront. The Ops teams should have extensive visibility into the testing cycle of new changes and how that works, and become the organization always asking Dev how it can “prove” its change works. The Ops teams must become the in-process auditors of the Dev teams. This keeps the Ops teams from being compensated for conditions that only exist at the end of the SDLC process, where they feel “dumped-on” by more rapid change delivery. The new comp model for Ops must focus on the number of changes successfully introduced into the production environment (i.e., the quantity of change that works), not just the stability they traditionally maintain.

    The Dev team, on the other hand, must begin to be compensated only for after-action success. Dev teams must look at their code after delivery to production—and only if the code is validated (i.e., it works), is adopted by its intended customers (a completely different angle) and performs within expected timing parameters (i.e., it scales well). This becomes the criterion that is the foundation of the Dev team’s measurement of success. This puts an Ops set of glasses on the programmer instead of making him blind to those business issues. The Ops teams become the key advisors to the Dev teams on how to prove quality. And the joint discussions lead to a more sensible monitoring strategy, which is needed across the board.

    There’s Compensation, and Then There’s the Bonus

    Changing the point of view for the compensation is only a start. Next up: Begin to alter bonus compensation to a more lean model. Instead of large annual bonuses, where the basis for achievements often are colored by my boss’s mood over the last six weeks, organizations should increase the frequency of bonus payouts to match the duration of epic completions—or, if possible, of sprints. The idea is to tie meaningful bonus payouts to the length of time it takes to produce only successful change. If the change works, the company offers additional bonus comp; if not, then the team must improve to secure the next bonus opportunity.

    Tie the bonus duration (whether to epics or sprints) to meaningful business impacts. In this way, each contributing employee understands “why” the work he or she offers has meaning to the company or its customers. This kind of bonus system allows for extreme flexibility in the priority of what teams work on throughout the year. Bosses can change priorities or shift them to match what executives see on the horizon. Employees are not tied to some list of annual work accomplishments that in a DevOps rapid world they had little shot of achieving unless they did them in a week. Instead, they have real-world goals, tied to real world time frames, where employees see the reward as they succeed at the work.

    Organizations can add even more bonus comp this way if over-and-above work is needed. And “if” organizations compensate both Ops and Dev only for successful change (which implies quality in the SDLC and stability in Ops) they create a new rewards system that enables adoption rather than repel it. Beyond finding the talent they need, organizations need innovative methods of retaining them. Substantial rewards in real time might just be one way to do that.

    Finally, being able to see the business impact of the change the joint effort of Dev and Ops teams produce allows for better ability to offer “appreciation” ideas as well. A simple “thank you” goes a long way in employee satisfaction, and when the executive and the team member understand “why” they are being appreciated, they go even longer. DevOps is not just a catalyst reserved to the “techies” who practice it; DevOps is instead a new way of thinking about how a company innovates. This kind of infectious thinking can easily spread to tertiary support organizations such as HR and Legal, if you let it.

    To continue the conversation, feel free to contact me.