devopsy.kinsta.cloud

Tag: adopting DevOps

  • DevOps: Build a Continuous Improvement Culture

    DevOps: Build a Continuous Improvement Culture

    DevOps was built around the concept of continuous everything. We even use it in many of the names—continuous delivery (CD), continuous testing, continuous integration (CI)—and it is a worthy goal to want to improve the places that form bottlenecks in your software delivery systems and processes.

    But what about the bottlenecks in the DevOps systems and processes?

    Like any other system, DevOps involves tools. We choose those tools, invest in learning those tools and use them to do great things. But then we have investment in the tool. If our knowledge and systems for deployment are all based around CD tool X or ARA tool Y, we are just as unlikely to want to move from it as we were unlikely to want to move away from the previous toolset.

    Even if you discount the number of shops that have done pilot or small-team DevOps but kept the core of their business waterfalling, there are still a large number of organizations that stepped into DevOps and stopped improving. Granted, it is a huge job to move a significantly sized application portfolio into the DevOps mold and toolchain, but it is contrary to the process/goals to stop just because you did.

    The market is nowhere near finished evolving: New tools are coming available and massive improvements to existing tools are being released. Keep taking advantage of those facts.

    The more projects/portfolios you have moved into DevOps, the more weight there is to resist change. That is the simple reality, just as the larger a rock, the harder it is to move (assuming the same type of rock, of course). The answer: Watch for those improvements that will make moving the rock worth the force required to do so.

    Working containers into the overall DevOps process, for example, can benefit every step in the toolchain. But if they’re not already in the process, it is a lot of work to get them folded in. I chose this example because for most shops, a little research will show it is well worth the effort. From preconfigured Dev/Test/Prod servers to run-and-done compile instances, containers can make the process more manageable and more agile. The benefits outweigh the costs for most shops.

    So keep an eye out. Don’t implement and spend all of your time on the tools you’ve chosen. Stay aware of what is out there, and what is coming. Understand how it would impact the DevOps process and try to pull in those items that fit the environment.

    And, as always, keep rocking it. The end goal is to keep the systems up, running and usable. And most of you are doing a great job.

    (This, and all blogs, brought to you as much by the geeks behind the scenes as the author and editor.)

    — Don Macvittie

  • The Past, Present and Future of DevOps Infrastructure

    The Past, Present and Future of DevOps Infrastructure

    Infrastructure automation enables organizations to set up operating frameworks that allow smooth functionality with enhanced usage, minimal scope for error and faster deployment of code. The process of infrastructure automation eases development and operations processes by introducing good deployment practices such as computerized testing and versioning.

    The need for automation was realized when the demand for deploying error-free applications at a faster rate increased. Additionally, competition from similar businesses forced organizations to look for a way to bridge the gaps between development and operations to deploy applications that would serve their business needs.

    The Challenge of Implementing DevOps Practice

    The following challenges present a roadblock in bringing out the best results of DevOps practices:

    1. The disparate work cultures of the development and operations teams.
    2. Non-involvement of DBAs in release cycles.
    3. Failure in deployment of error-free codes.

    The Emergence of DevOps

    The introduction of DevOps practices resolved a lot of problems in terms of increasing business operations efficiency. It created a common operating framework where disparate teams including development and operations teams can work together and deliver applications faster that will, in turn, result in effective business operations and an increase in ROI.

    Adopting DevOps involves:

    1. DevOps experts outlining of traceable metrics for creating a roadmap by visualizing the desired outcome.
    2. Creating a pilot framework that will include the tools of both development and operations teams further integrating these existing tools with the DevOps experts’ robust ecosystem,
    3. Analyzing, designing, constructing, automating and implementing the network based on the metrics and resources collated.

    DevOps Tools:

    The DevOps infrastructure automation process uses several tools that define and drive the success of the implementation process. Here is a short overview of some of the more well-known and widely used tools.

    • Jenkins – A Java-based continuous integration (CI) tool contributes to the faster delivery of applications. Jenkins must be aligned with a version control system such as GitHub or SVN. Whenever the code repository receives a new code, a Jenkins server builds and tests the new code and informs the team about results and any changes. Over time, Jenkins has moved from being just a CI tool to being used as an orchestration tool that builds pipelines for application provisioning and deployment.
    • Chef – A ruby-based configuration management tool, Chef includes cookbooks that code infrastructure in domain service language (DSL) in combination with a little bit of programming. Chef creates virtual machines and customizes them in accordance with the rules listed in the cookbooks. An agent residing on all the servers that need configuration extracts the cookbooks from the chef master and runs them on the server to get the desired results.
    • Puppet – Also, a ruby-based configuration management tool, Puppet was developed with system administrators in mind. A configuration code is written using Puppet DSLs and is wrapped in modules. Puppet performs a similar function to Chef but, unlike Puppet, Chef is a developer-focused tool. It runs a puppet agent on all servers that require configuration and extracts the compiled module from the puppet server and installs required software packages as prescribed by the module.
    • Ansible – An agentless configuration management and orchestration tool, Ansible can be used for cloud computing. Ansible has configuration modules called “Playbooks,” which are written in YAML format and are easy to compose in comparison with other configuration management tools.
    • Docker – Docker, a process-level virtualization tool, creates isolated environments for applications known as containers. These containers can be transferred to any other system without making any application changes. Docker has evolved as the next step in virtualization. It is beginning to become a favorite of many DevOps practitioners and leading cloud experts.

    The Future of DevOps Infrastructure Automation

    A lot of organizations are beginning to benefit from the implementation of DevOps infrastructure automation. And with the pace at which the demand for DevOps services is increasing, infrastructure automation has a long way to go.

    As per statistics, by bringing DevOps practices into the picture, organizations can benefit by deploying code up to 30X more frequently and reduce the deployment failure to less than 50 percent.

    — Veritis

  • Is DevOps Really Necessary?

    Is DevOps Really Necessary?

    I spend my days discussing DevOps and digital transformations with companies of all sizes, from startups to some of the world’s largest enterprises. I’ve found that many teams are still at a very early stage in modernizing their technology, culture and practices. However, I’m surprised at the number of the organizations that don’t believe in change because they “have been doing just fine up until now.”

    This is a recipe for failure in the long term. Their competitors will likely be working in loosely coupled teams with shared responsibilities and increased collaboration. They will have adopted continuous integration, continuous deployments/delivery and automation, to name a few processes. In time, these competitors will begin to consume the laggards’ market share. To put it into perspective, you might still be talking about your next deployment coming up in six weeks time, while they are talking about deploying into production 12 times today alone.

    With their increased agility, stability and reliability, these more successful teams will be faster. They will not be afraid of failure and will have built fault-tolerant systems that can rapidly and automatically repair. These self-healing, well-tested systems and processes will free time for developers and other team members to focus on developing newer and better applications in response to customer feedback—all while the your team might continue “doing fine.”

    We aren’t just talking about buzzwords here; the data shows that those who are playing catchup are also potentially causing themselves more harm than good. Puppet’s “2017 State of DevOps Report” shows that the gap between higher- and lower-performing teams has narrowed as throughput has increased, meaning lower-performing teams are working faster, shipping more code to production and responding to changes quicker than they were the previous year. This is great in principle, but in 2017, change failure rate had also increased, showing that low performers were actually more likely to fail when making changes than they were in 2016. The data shows that low performers had a 3x change failure rate in 2016, which increased to 5x in the 2017 Puppet study. From a business standpoint, this means that those higher-performing teams are able to deliver more value to their customers by responding faster and more reliably to feedback and market demands.

    Companies should certainly be paying attention to what their competitors are doing, no matter what area or sector in which they operate. However, they also must not lose sight of the importance of quality, security or feedback during this journey. If you know you are at a much earlier stage in your transformation, focus on getting it right. There will always be setbacks and challenges, but as the old saying goes: This is a marathon, not a sprint.

    DevOps methodologies, when applied effectively, will make your organization more successful.

    — Conor Delanbanque

  • Unnatural DevOps Delivers Supernatural Results

    Unnatural DevOps Delivers Supernatural Results

    DevOps is not necessarily a “natural” process. Asking people to function outside of their comfort zones and prescribed roles is a challenging task for any organization, and requires a shift in organizational behavior. At Trace3, we believe that while DevOps is not natural, a supernatural result is achievable with the right cultural transformation supported by good tools that drive collaboration and automated flow between people and tools in the application delivery workflow.

    We define DevOps as the application of the principles of continuous flow, feedback and improvement to people, process and technology for increasing agility, stability, security, efficiency, quality and satisfaction. Or, to quote from Michael Schmidt’s article, “Ten ways DevOps will benefit your IT department“: “When implemented properly, DevOps can transform your software engineering process and create value for both employees and customers, producing business performance and value.” These are huge “supernatural” benefits that every business would like to achieve.

    Implementing DevOps properly is challenging. DevOps depends on collaboration; smooth process flow from conception through deployment; and feedback between people, processes and technologies. It’s more natural for people and departments to hoard information and centralize control—traits that are contrary to DevOps best practices. People tend to trust their own abilities and are skeptical when they need to trust dependencies on others. People view competition with others as a means to personal advancement. Compounding this, organizations often manage, measure and reward success at the silo level instead of recognizing a successful delivery as a result of successful execution across the end-to-end DevOps value stream.

    Breaking down silos through transparency and the free exchange of information helps organizations implement the unnatural DevOps processes needed to achieve supernatural results. Fortunately, the transformation is assisted with technology. For example, continuous automation tools liberate sharing of infrastructure, application and compliance information. Pauly Comtois, Hearst’s vice president of DevOps, describes the role of strong tooling in successful DevOps transformations in his article, “Enterprise DevOps at Hearst Digital Media“: “Tools can be an incredible asset in this regard. We deploy Chef across multiple teams in the value stream, automating the deployment, configuration and state of environments in Dev, QA and Production.”

    In scenarios such as the example above, tools support integrated and automated processes for builds, configuration, infrastructure and deployment management. Information managed by continuous automation tools is available for DevOps uses by providing a single “public yet controlled” source of truth for configuration data useful by all processes and tools in the toolchain across the end-to-end pipeline. Those who were concerned and hoarded configuration data can trust that the tools will keep the information secure, reliable and available on demand as readily accessible code.

    Moving beyond the natural behavior of organizations and individuals is difficult, but success can create supernatural results for long-term growth and team effectiveness.

    — Marc Hornbeek

  • Sometimes It is More Than Culture, It’s About Gains

    Sometimes It is More Than Culture, It’s About Gains

    Almost since the beginning, there has been a tug-of-war between whether better toolchains or increased communications were “real” DevOps. The answer to that question is, Yes.

    Better tools undeniably speed the delivery process and streamline operations. Anyone who claims differently is not living in the real world. Jenkins alone can make your development world a better place, and it is but one tool—with hooks into like a million more, but still one tool.

    Increased communications speed the delivery process and streamline operations, if they are frank and honest. A developer who knows that his choices for included dependencies can make life easier or more difficult on maintenance is far more likely to include that consideration in their choice, for example.

    Gains Are In the Doing

    But the rock-star gains that some shops have seen from DevOps? Are a bit of both, and something else. The sudden improvement of 4x delivery frequency most often comes from no longer avoiding that thing you don’t talk about, or the legacy piece of software that eats an increasing amount of time just being kept alive because it’s necessary.

    And I would argue that is the best place to start. I’ve touched on this tangentially before, so I thought I might be more direct.

    You have it. That thing in the process that is a bottleneck. No one talks about it. There are reasons—some because a few years ago you tried to resolve it and couldn’t; some because the bottleneck is a natural outgrowth of some requirement; some because “it’s always been that way,” But there is that step (or more often three or four steps) in the develop/test/deploy cycle that consumes a huge chunk of time to do little work.

    There are also those tools/processes so old that they consume an inordinate amount of resources. Most shops more than a few years old have them. I’ve worked in a couple of 100-plus-year-old companies and frankly, these bits of outdated software were part of the genotype of the org because they’d been in IT so long.

    Manual processes are known to take time, simply because they’re manual. Development itself, even when broken into smaller pieces and implemented piecemeal in the manner of Agile, still takes time. Testing, too, is still (or again, depending upon your background) a largely manual process. And security checks are also in a ton of shops.

    These are known. And if you’re approaching the problem from the “find the bottlenecks,” I would exclude them. Better tools or more automation can help all three, streamlining who does what can help test and security, but these aren’t the places that I’m thinking of.

    Aging Support Software

    I worked in a shop once that spent an inordinate amount of time keeping a “database” alive that was in existence before RDBMS. Yep. There was mission-critical software that used the database, so it was maintained as well as possible internally. Internally because the vendor no longer existed.

    Several analyses were done, and each time it was decided that the DB couldn’t be gotten rid of without a major rewrite of the systems it supported. But as time went on more and more resources were poured into those systems and the database. Eventually, it was forced from above that the database had to go. The estimate to do so came in at an insane number and the project was dropped. Meanwhile, an increasing amount of resources were being poured into the DB and the systems that supported it. All were outdated, and all required special skills that were also outdated. Soon the org was paying exorbitant rates for the few people who could work on the system to do so via contract.

    Finally, with hard dollars attached to the preservation of this system, a forced upgrade was a viable project, and it went forward.

    This is all pretty standard enterprise stuff, it’s kind of the way IT has functioned in the past. But had IT management gone to business units with improvements that could be offered during the upgrade, all of the resources poured into not upgrading over the years could have been used for new product/feature development. Identifying the fact that replacement was inevitable and wouldn’t get cheaper for waiting is the type of outdated software issue I’m talking about.

    Bottlenecks

    From the bottleneck side, there is often a “this monster log file must be reviewed by a human” step on the way to production, too. Think about simple automation. What is the human scanning for? Talk to those responsible and see if they can offer guidelines concerning how they would train a new person to review the file. Then write a small script to do it—not as a final solution, but to show that a person doesn’t have to if the review is well-bounded. People won’t trust it, so run it in parallel for a while, and learn from what it misses, point out what it finds. Long term, eliminate the delay of someone pouring over logs looking for the same things over and over. Take time to make it possible. It won’t be a ton of time, but it will reap gains. Just talking to the team that must accept the log output might reap gains.

    There isn’t a need to immediately install 15 DevOps tools or reorg all of IT before starting your DevOps journey. Just look for the places resources are being invested to little long-term gains, or where the process is held up in a manner easily eliminated. And keep rocking it.

    — Don Macvittie

  • NetOps vs. DevOps: New Data is Enlightening

    NetOps vs. DevOps: New Data is Enlightening

    Traditionally, we have seen development—and, by extension, DevOps—as having a less-than-cordial relationship with network operations (NetOps). Since the goals of the two groups are sometimes at cross purposes—DevOps wants access to make everything run automated, and NetOps wants to control a production environment so that it is stable—we’ve naturally talked in those terms.

    I’m a developer who learned other skills, including operations, along the way. I can manage to get things done on F5 Networks or Cisco Systems gear, but I’ve sat with true experts while they worked their magic and have a lot of respect for the NetOps crowd. So this topic has always been of interest to me.

    Now, according to a recent survey conducted by F5 Networks, we can see that the animosity we have all been lead to believe is there really isn’t. There is also a press release here, and surprisingly, it offers more information from the survey than the blog does.

    There is a ton of interesting data about rate of automation, views of other teams and deployments in the survey results, making it well worth a read-through.

    While DevOps and NetOps remain at odds regarding how much pipeline access should be available via self-service and automation, for the most part both groups view the other as on the right track. Eighty-two percent of DevOps and 76 percent of NetOps agree that each other prioritizes “the right things.” Clearly, there is common ground across the digital divide in at least goals and focus, if not always in the organizational chart.

    This, to me, says that DevOps is having the proper impact on the organization. That what’s important is getting prioritized on both sides of the pipeline. Or it says that this was always the case, and DevOps has had little impact. Historical surveys, however, seem to indicate that there was not always this level of belief between the two groups.

    Interestingly enough, there has been a lot written about whether your DevOps methodology should be operations- or development-driven, but networking has always been a bit off to the side. Not the type of operations that historically is discussed in DevOps; it sat with teams such as Security and QA working with DevOps but not generally a part of the DevOps teams.

    The results here would indicate that network automation is occurring at a healthy clip (makes a guy want to look at vendor APIs again, and see what’s changed to enable this fact), and if the majority of DevOps people are saying the NetOps team is focused on what’s important, likely the integration into DevOps is further along than pundits have portrayed.

    Though the results indicate that many DevOps teams want more access and control of network operations than they have, this is likely the natural outgrowth of the different views these teams have always had. Network operations’ goal is rock-solid infrastructure. DevOps’ goal is speedy delivery. These two things can be at odds relatively easily.

    I’m hoping that F5 offers us more access to data than a blog and a press release, but honestly what is in those two is enough to get you pondering about the state of NetDevOps (or whatever we’re calling it this week) versus the perception.

    Disclaimer: I am a former F5 employee, and have a positive view of the company and its products. My wife is also F5’s Principal Technical Evangelist, and had a hand in this survey. While this doesn’t impact writing about survey results, you’ve been informed.

    — Don Macvittie

  • To DevOps Or Not To DevOps

    To DevOps Or Not To DevOps

    Although enterprises today all claim to be software companies, there remain four distinct groups of organizations: the technology leaders, the technology followers, the technology hopefuls and the technology laggards.

    The first group are the industry leaders who are paving the way. These are the organizations coming up with new technologies, processes and cultural shifts. They write articles and books and present at conferences, and despite perhaps working at smaller companies, they are the names you may recognize. These are the companies the rest of us aspire to be—the Googles, the Amazons and the Microsofts that keep the rest of the industry hanging on to every word.

    The technology followers typically are smaller organizations that tend to adopt the latest and the greatest technologies. They are the leaders’ first wave of followers who are not afraid to try new things, nor afraid to fail. They have cultures that allow them to experiment and adapt and are given a lot of free time to invest in learning and implementation, after they watch the big guys successfully implement.

    However, the first two groups are the minority of software organizations, since not everyone can lead the way, nor can everyone have the luxury of spending a significant portion of their time experimenting with new methods. After all, most of the new technologies do not have a long lifespan. They appear, generate some attention and then disappear.  Front-end frameworks provide a great example of this: new ones pop up and exist for a few months before being superseded by other frameworks.

    Another example of technology followers is containers—few doubt their usefulness and most of us are convinced they are the best solution. The only question is which cluster orchestration framework to choose. These frameworks are evolving so rapidly, with new ones introduced on a weekly basis, that it’s almost certain most will be defunct within a year. So, if you are into experimenting with the latest offerings, chances are you could make a wrong choice. Will it be Kubernetes? Docker Swarm? Mesosphere Marathon? Rancher OS? Cloud Foundry’s Diego? CoreOS Fleet? The question is not whether to adopt one of those tools, but rather, which one has the best shelf life.

    Most of us fall under the third group, the technology hopefuls. We don’t have the ability to lead the way, nor do we have the luxury to explore the latest and greatest technologies. Yet, we cannot allow ourselves to become obsolete. The industry is moving so fast that just a few years can leave you at the back of the pack where no effort or investment can get you back in the game.

    The technology laggards are the software companies that have fallen behind. This group includes businesses that are struggling to remain intact and run the risk of disappearing. These are the organizations that do not innovate, do not adapt, do not invest (enough) in improvements and do not even understand the current industry trends. They are the organizations that do not comprehend that becoming obsolete doesn’t take decades, but years—sometimes even months.

    Based on the definitions of these groups, chances are, unless you are Google or Microsoft, you are not in the first or even second groups. By reading this article you are making the effort to be informed so you don’t lack the self-awareness that places you in the fourth group. Like most software organizations, you fall into the third group of technology hopefuls.

    As part of this group, you have to balance between the latest and the greatest technologies and the chance of becoming obsolete. You cannot try every new solution that comes along, nor you can let years pass by until you jump on the train. The big questions you have to ask are, How can I distinguish what will stay and what will disappear? How do I make a choice between what to adopt and what to ignore?

    There may not be a one-size-fits-all answer, but one process you absolutely cannot ignore is DevOps. If you’re asking yourself whether your organization should adopt DevOps, the answer is a resounding “yes.” And I encourage you to adopt it now, before you are replaced by other companies that are already living and breathing DevOps. Otherwise, you’ll be unable to adapt to the speed needed to keep up in today’s market and understand the cloud and scale accordingly when an avalanche of users come to your site after a successful marketing campaign. Lacking preparedness could cost you your business, especially in a competitive market.

    DevOps is here to stay, evolving similarly to Agile by providing value and answering many of today’s challenges. The only problem is that adopting DevOps can be difficult. To implement DevOps, you might have to change your company’s culture; you might need to adopt new tools; you might need to change processes; and, you might need to change your software architecture. Changing any of those aspects is hard, but changing all of them is very hard.

    Your culture and company structure need to be agile (in the literal meaning of the word). The old, monolithic tools need to be replaced with smaller, much more dynamic ones that will work well with the ever-changing nature of cloud infrastructure. Processes need to be automated, since there is no time to pause for manual intervention. To accommodate the needs of automation and frequent deployments, your applications and services will need to be relatively small for continuous delivery or deployment to become a standard process. Your teams need to be highly specialized and fully capable of working on all aspects of the software development life cycle, from requirements to post-production support.

    It’s a harsh and challenging road ahead, but at the end of it lies success. You’ll experience faster time to market, greater reliability, high availability and faster adaptability to market changes. Some are already there, while others already embarked on the journey. Don’t be left out—the alternative will leave you obsolete.

    To DevOps or not to DevOps is not the question. For most organizations, adopting DevOps is the answer.

    About the Author / Viktor Farcic

    viktorfarcicViktor Farcic is a Senior Consultant at CloudBees, a member of the Docker Captains group, and books author. He coded using a plethora of languages starting with Pascal (yes, he is old), Basic (before it got Visual prefix), ASP (before it got .Net suffix), C, C++, Perl, Python,ASP.Net, Visual Basic, C#, JavaScript, Java, Scala, etc. His big passions are Microservices, Continuous Integration, Delivery and Deployment (CI/CD) and Test-Driven Development (TDD). He published “The DevOps 2.0 Toolkit: Automating the Continuous Deployment Pipeline with Containerized Microservices,” and  “Test-Driven Java Development.” He is currently working on his third book, “The DevOps 2.1 Toolkit: Docker Swarm.” Connect with him on Twitter.

     

  • The Forgotten Fourth Question

    The Forgotten Fourth Question

    A very well-written paragraph about DevOps appears in Mike Loukides’ book, “What is DevOps”:

    … Modern applications, running in the cloud, still need to be resilient and fault tolerant, still need monitoring, still need to adapt to huge swings in load, etc. But those features, formerly provided by the IT/operations infrastructures, now need to be part of the application, particularly in “platform as a service” environments. Operations doesn’t go away, a good part of it becomes part of the development. And rather than envision some sort of Uber developer, who understands big data, web performance optimization, application middleware, and fault tolerance in a massively distributed environment, we need operations specialists on the development teams. The infrastructure doesn’t go away – it moves into the code; and the people responsible for the infrastructure, the system administrators and corporate IT groups, evolve so that they can write the code that maintains the infrastructure. Rather than being isolated, they need to cooperate and collaborate with the developers who create the applications. This is the movement informally known as DevOps.

    These days, recruiters to managers, engineers to analysts and, of course, CXOs all are interested in DevOps. They have questions such as, “What is DevOps?” “What are the DevOps requirements?” “What are DevOps implementations challenges?” But the key question is, “Why does DevOps remain difficult to implement?”

    Adopting DevOps does not simply mean adopting new processes or toolsets. “Why DevOps” can be understood better by focusing on the benefits it will give you. So let’s dive deeper into one of the key best practices we discovered in our journey into DevOps.

    Agile to DevOps

    Business challenges, complexities and competition pushed companies to innovate and solve their existing business problems in a better way, adding more features and integrated solutions. Ways of implementing solutions vary from organization to organization, adoption of tools and technologies also differ by small and large organizations based on complexity of problems, but it’s clear that many have adopted agile practices in the past two decades. Agile was made clear by a few well-defined processes such as iteration planning, daily scrum meetings, scrum of scrums (in larger enterprises), iteration demos, retrospective meetings and small, empowered teams.

    One of the key mindset changes most of the IT world went through while transforming themselves into agile execution was to have more verbal communications and daily scrum meetings. Daily scrum meetings required the scrum master to ask three questions:

    • What was completed yesterday?
    • What will be completed by end of today?
    • Any blockers?

    These three simple questions kept things simple and quick.

    As agility is extended to Ops now, there is a need to revisit whether these three questions are enough. There is a need to revisit the attendees in scrum meetings too. There is a need to add some more responsibility to the role of the scrum masters’ communication radius.

    Before we look at the fourth question, lets let look at the key reasons to adopt DevOps implementations in organizations:

    • Achieve higher speed of time to market without compromising Ops stability
    • Better resource utilization across Dev, Test and Ops
    • Be more responsive to customers

    Why would speed of delivery hamper stability in Ops? In our firsthand experience, the communication between Dev and Ops is not that rich in most of the organizations today. If one studies the root cause analysis of problems faced by Ops in trying to keep the environment stable and healthy, one will find some if not all the following issues:

    • Developers are not deployment-aware
    • Developers do not think of exposing or exporting the right controls for allowing Ops to configure app dynamically (Unintentionally. They simply don’t know.)
    • Many configurations are embedded deep inside the app
    • Adding specific third parties, changing their versions is sometimes missed in Ops communications
    • Ops is unable to clearly articulate what they require from Dev, even when everyone is co-located
    • Continuous firefighting in Ops results in fast fixes, better debugging, but not proactive processes
    • Even if weekly meetings between Dev and Ops happen, they do not have a clear agenda or metrics

    While there are many more symptoms and their causes, and while there are multiple solutions to fix processes and problems, we have found particularly one solution that seems effective.

    We added a fourth question to daily scrum meeting. That fourth question is: “Are you blocking Ops?”

    This question is a direct question to each developer in daily scrum meetings. This question essentially means to ask:

    • Have you written in any code that requires a new configuration?
    • Have you written any code that requires change in existing configuration?
    • Have you written any code that is creating a new binary or file that needs to be installed in production?
    • Have you recompiled existing code with new third party libraries because older ones are unsupported, even though there may be no other change in functionality?
    • Have you hard coded any IP address or other configurations into the code which cannot be changed without rebuilding?
    • And so on…

    Ask the Fourth Question

    The above list of questions is developed based on the root cause analysis of app stability in Ops. This list will be different for different developers/teams. The scrum master communicates the appropriate actions to developers based on the answers to these questions. The scrum master also alerts the Ops team and gives them a heads up on what’s coming their way, and provides timely feedback to the developers.

    Is this question important enough to be the fourth question in scrum meetings?

    Or is this question covered as part of the “Any blockers?” already being asked? One can debate either way. The blocker question in scrum meetings is thought of in a limited Dev/QA scope, and that scope has not included the Ops team until now. Given that Ops team faces customers, given that the DevOps movement is about improving synchronization between teams, this Ops question needs to be a first-class citizen in daily scrum meetings.

    Lastly, it’s critical to to look at root cause analysis-related Ops problem as early as possible. The scrum master becomes more mature as he asks this question to his team while interacting with Ops on hearing them out. If teams are funded enough to have someone from Ops into the scrum meetings, then he/she can be the primary representative for Ops and can steer the delivery to be Ops-friendly.

    Irrespective of how one achieves it, taking care of configuration issues proactively in daily meetings, and thinking about Ops on a daily basis in Dev meetings is key to successful DevOps adoption. Configuration issues are not the only focus, but they do form a large chunk of Ops issues.

    From our metrics, customers have seen following benefits after adding the fourth question their scrum meetings:

    • Shipping code 7x more frequently
    • 45 percent fewer failures
    • 12x faster ability to recover earlier in the tool chain

    Please feel free to share your findings with us @ info@opexsoftware.com.

    Happy DevOps!

     

    About the Author/Sanju Burkule

    sanjuSanju Burkule, a pioneer, visionary and creator of Products in the DevOps Space @OpexSoftware. He writes about automation and strong, connected, integrated IT and tweets often about these topics.

     

     

  • Measuring the Cost of Service Creation & Adoption

    Measuring the Cost of Service Creation & Adoption

    Adopting and implementing DevOps isn’t an overnight project. It takes time and effort to determine how it best fits into your organization—and what benefits your organization will realize as a result.

    Creating an enterprise-class DevOps service is typically done in one of three ways, depending on the organization, its leadership and its employees. Each has its pros and cons, and each has a number of factors to consider.

    A Fully Centralized Model

    In this model, tooling decisions are done in a central service provider group for the entire organization. Tools are managed centrally and standards are created centrally. DevOps automation creation (at least the templates or frameworks) is done centrally. This model is highly advantageous for organizations with a wide variety of technology platforms (think large population of legacy apps) and use many technology components for their apps, such as multiple types of databases, middleware and operating platforms, for example.

    Not many large organizations are ready to scrap what they have in the application portfolio and start over using the cloud and containers and develop in a singular methodology. Using a centralized model, companies can continue to improve upon their years of investment in legacy applications and adopt a DevOps continuum without starting over from scratch.

    This also is the lowest-cost model, as tooling is standardized and administered from a central point and licensing can be negotiated to maximum effect. In addition, labor (generally the highest ongoing cost of any service creation and implementation) is minimized by centralizing it. Training costs also can be minimized by establishing a centralized training program for all impacted audiences. Companies can change their incentive programs to achieve hiring objectives and business results as priorities change—something that isn’t practical in a decentralized or federated model.

    A Fully Decentralized Model

    In a fully decentralized model, tooling decisions are determined and managed by each development team. This has the advantage for shops with small numbers of applications or singular methods of development—fewer applications are better candidates for starting over with cloud or container integrations and can take advantage of the newest technologies from the starting point, if needed.

    Shops such as Etsy or Facebook seem massive, but from a user point of view, they really only present a single application that needs to be maintained and improved over time. Instances like this may favor a decentralized model because the actual number of application candidates may be highly limited—sometimes only one.

    This model typically represents the highest costs, as repeatability and standardization can only be measured at the development team level. As such, this is practical for a very limited number of applications, or technologies; if those numbers grow, the strategy quickly becomes untenable. In addition, this strategy has an uncanny history of pulling development resources to do the DevOps automation engineering that needs to be created and maintained. Typically, the highest skilled developers are assigned these types of duties, and thinking in terms of templates and frameworks is rarely their first approach. Developers tend to be expedient, and believe they will always be there to make sure it works. This also can reduce the potential for repeatability and predictability (raising costs).

    A Federated Model

    In a federated model, tooling decisions and standards are done centrally, but management and operations of the tools are performed by a limited number of subservient organizations (often by line of business). Centralized templates may or may not be viable depending on the variability of the business units they serve. The chief advantage of this approach is the segregation of tooling and reporting by business unit (typically) without having to build that kind of segregation in to tools that were not designed that capability in mind.

    This model also has the advantage of increasing the speed of adoption, as several parallel efforts can take place at the same time and spreading engineer resource demand across multiple organizations. It typically makes the most sense in larger organizations with multiple business units whose application portfolios that would benefit from an onboarding to DevOps.

    What’s more, a federated model may be the most cost-efficient. Labor costs will be inherently higher as functions are replicated from business unit to business unit; however, that replication will result in increased adoption rates and project completions (you get what you pay for). If business units are able to leverage centralized templates for each technology type, adoption rates become an order of magnitude faster (or have that potential). Tooling costs still can be negotiated centrally, even though management will occur in the business units. Again, there is hardware replication from business unit to business unit, but the ability to segregate users, permissions and operations down to the business-unit level can save tremendous labor costs if development was the only other option to create that capability in tools not designed for it.

    Adoption Costs

    Measuring the cost of adoption requires focus, no matter which method you employ to create the DevOps service. Applications moving to DevOps are said to be onboarded (where the original build, deploy, test and release automation is created). To successfully onboard an application requires education of the sponsoring executives, training of the impacted developers and testers, and coordination with the change and release managers.

    Collection of meta data about the behavior of application components is no small task, requiring frequent follow-ups and action items to reach completion. The effort to onboard applications to DevOps becomes the first cost for estimation and tracking as the application portfolio is attacked.

    Even after onboarding of a given application is complete, costs continue. The terms “continuous delivery” and “continuous integration” should also infer that application development methods “continuously” evolve and may adopt new technology components. Significant changes may occur that require revisiting the DevOps automation to adapt to these ongoing revisions and updates.

    While this work is rarely as extensive as the initial onboarding, it is measurable. Work of this nature is usually referred to as BAU (business as usual) and represents the second significant cost of DevOps services that must be measured and tracked. If no thought is given to this work effort, inevitably the team assigned to do onboarding will be distracted by the BAU requirements until they reach the point where no new applications can be completed.

    Staffing cost estimations can be presented at the component level for both onboarding and BAU efforts. For example, if one DevOps engineer can onboard 20 components per week (assuming familiar technologies within a given portfolio), and if the general ratio of components per application is 4:1, then a given DevOps engineer can successfully onboard five applications per week. If the engineer is never distracted with BAU, this can continue until the entire portfolio is onboarded. However, my experience has shown that for every 20 applications completed with onboarding, the BAU work effort can absorb nearly a full DevOps engineer. So if you start with a team of 10 people, you will onboard only 200 applications total before the entire team is required to focus on BAU work effort and has no more bandwidth to continue onboarding new apps. Understanding this demand, and learning how efficient your particular engineering team is, will be key to managing costs while completing the onboarding efforts of large application portfolios.

  • DevOps Stack on a Shoestring Budget

    DevOps Stack on a Shoestring Budget

    There are competing products, both open source and licensed, in any DevOps area that you can think of. The Operations groups in companies tend to accumulate such applications over time as determined by ongoing projects. Team members’ familiarity with a certain application could be a factor in adding it to the tools stack.

    Companies, therefore, end up with a mixed bag of freeware, open source or community versions of applications and software licensed from the get-go. In a startup environment where the Operations group might be on a shoestring budget, it might make sense to build the DevOps infrastructure on the cheap; using open source software (OSS) products would be one way to achieve that goal.

    The Shoestring Budget

    Below is a list of some of the applications and software for an effective DevOps environment. The list here includes only core applications that are needed to build the DevOps infrastructure. The basic communication and productivity tools that would be used by everyone in the company—and usually managed by IT—are not in the scope. Neither are the compilers and debugging and testing tools used by product development teams. However, in a small company setting, managing all kinds of tools might drop into the laps of the same people driving the DevOps initiatives.

    The objective here is to make a list of software that could be used license-free. That means some of the obvious applications in certain areas might not make it to the list. That doesn’t mean that the open source alternative is an inferior choice; availability of reliable product support is a major factor in most of the software purchase decision-making processes, and licensed software scores over OSS on that front.

    Git for Source Code Management (SCM)

    A code repository might be the first DevOps infrastructure component that would be set up even before a product idea is formalized into a company. Also, both developers and DevOps engineers need it for collaboration, within their own teams and across various engineering teams. Typically in a startup, developers get started with the DevOps work such as build and deployment automation before a dedicated DevOps team would even be part of the engineering staff.   

    In this category, It is hard toimagine choosing anything other than Git. Even though GitHub is not free for maintaining private code repositories, Git server can be setup on-premise for free. AWS has CodeCommit, another SaaS version of Git, as one of the services, and there is no extra cost for using it if you are already on AWS.

    Bugzilla for Bug Tracking

    The popular Atlassian suite of products, JIRA in this category and Confluence for collaborative documentation, seems to be default choices in respective categories. But, both on-premise and SaaS offerings of  JIRA and Confluence are licensed. Bugzilla is a powerful OSS alternative to JIRA that can support a large user base and it has been around for a while.

    If you are open to customizing Bugzilla for specific requirements, it could work better than some of its licensed competitions. Though its primary use is bug tracking, Bugzilla is feature-rich enough to be used for managing all phases of software development and even as an IT ticketing system.

    MediaWiki for Collaborative Documentation

    Though there are multiple, open source wiki applications available, MediaWiki might be the best supported when it comes to the availability of third-party extensions. Out-of-the-box features of MediaWiki are limited, but it can be extended using the free extensions available. Such customizations are very straightforward.

    Jenkins as Build & Continuous Integration Platform

    Though there is an enterprise offering of Jenkins from CloudBees, its open source version combined with freely downloadable plugins would make up a powerful integration platform for build, release and deployment automation—and any other areas in between. Describing Jenkins any more here would be simply preaching to the choir.

    Artifactory for Artifacts Management

    This is one of the applications a lot of companies might not have in their DevOps stack in the initial stage. However, the need to store built artifacts and to cache third-party tools is real, and the packages and binaries would normally end up in S3 buckets or shared filer volumes. There is no reason for not using the OSS version of Artifactory right from the beginning, as it covers the storage requirements already mentioned and much more:

    • Automated uploading of artifacts as part of the build process, typically done from Jenkins
    • Caching third-party tools locally for builds and deployments
    • Storing software artifacts for meeting build dependencies
    • Distributing packages and third-party tools for deployments

    Ansible for Configuration Management and Orchestration

    There are two great products in this space that could be used for free: the OSS versions of Puppet and Ansible. If you are on AWS, Chef is also available, at no extra cost, in the form of OpsWorks. But I would recommend Ansible for those getting started for these reasons:

    • It is an agentless tool and beginners would feel safer to roll out automation using it. A configuration management agent can play havoc with production systems if something gets misconfigured inadvertently.
    • Using the playbooks, deployments can be orchestrated very well using Ansible. In many companies, Puppet or Chef primarily covers system management aspects and there would be additional tools for deployments. Ansible simplifies that scenario.
    • Ansible’s support of Python turned out to be more useful, in my experience, than the support of Ruby in Puppet and Chef. The chances of a DevOps engineer knowing Python is much more than Ruby.
    • Ansible’s handling of host inventory is advanced, especially the dynamic generation of inventory lists. The latter feature is compatible with the provisioning of cloud infrastructure where its elasticity demands the use of dynamic inventory lists in configuration management systems.

    Nagios for Monitoring

    A lot of us in the DevOps community despise Nagios for its blandness and lack of features out of the box. But if you are willing to customize it by developing application-specific plugins, it could be converted into a very effective monitoring platform that can be used to consolidate outputs from the rest of the monitoring and various admin applications. I recommend it for these reasons:

    • Writing plugins for Nagios is very simple. Also, there are tons of free plugins available that would meet most of the system-level and third-party component level monitoring.
    • Developing plugins to add application-level monitoring and to integrate with third-party admin applications is easy.
    • When you want to move on to a next generation cloud-based monitoring application such as Datadog, existing Nagios plugins are normally migrated natively or as an integration.

    In a sysadmin environment where coding skills would be in short supply, Nagios could be of limited use as a systems monitoring application. But DevOps teams should treat it as an easily extendable tool that can be used to implement specific and fine-grained monitoring goals of the company.

    ELK for Log Management

    Splunk was instrumental in changing the way we dealt with log files, and, now there are several good products including Loggly, Logentries and Sumo Logic in that space. Though some of these SaaS products have a free tier and that might be enough for a small operation, it is better to start with ELK (Elasticsearch, Logstash and Kibana stack), mainly because the size of the logs can grow very fast and so would the usage of the log management application in the company. ELK requires substantial DevOps resources to set it up and maintain, as there are three applications involved.

    Closing Remarks

    It is important to have these essential applications, or similar ones with comparable features, implemented or made available to empower the engineering  staff. They facilitate the application development, lay the foundation for automation initiatives, and provide the toolkit to maintain production environments 24×7.

    Though it is normal, the organic growth of the DevOps organization and infrastructure can bring in OSS products with overlapping features, mainly because they could be used license-free. Carefully selecting the tools would help avoid the otherwise-inevitable clutter and the confusion later.

    One aspect not discussed is the total cost of use of a software. If SaaS offerings such as GitHub, Datadog and Loggly, and hosted versions of JIRA, Confluence and Artifactory are used, the cost would only be the subscription charges. OSS applications will need computing resources and some amount of customization and administrative overheads, which could add up to the overall cost.  

    If the objective is to minimize total cost, the upfront decision-making can become much more complicated than it looks. However, for a startup or a small company, getting started with the OSS stack for setting up the DevOps infrastructure can be advantageous if the applications are selected carefully.